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DEFENSE  SYSTEMS  MANAGEMENT  COLLEGE 


STUDY  TITLE:  MANAGEMENT  OF  NAVY  TEST  AND  EVALUATION  MILESTONES 


STUDY  PROJECT  GOALS: 

To  identify  the  DoD  and  Navy  requirements  which  dictate  the  need  for  intensive 
management  of  test  eind  evaluation  milestones,  eind  to  define  the  framework  of  a 
computerized  data  base  of  milestone  information  which  can  be  used  to  develop 
structured  information  status  reports  for  senior  management  personnel. 


STUDY  REPORT  ABSTRACT: 

The  report  briefly  identifies  the  requirement  to  establish  viable  test  and 
evaluation  monitoring  during  the  acquisition  of  major  systems  and  equipment, and 
recognizes  the  significance  of  the  test  auid  evaluation  effort  as  a prerequisite 
for  successful  Defense  System  Acquisition  Review  Council  (DSARC)  decisions  and 
the  further  commitment  of  development  and  production  funding.  It  further  recog- 
nizes that  organizational  commanders  now  require  a more  integrated  overview  of 
command  test  and  evaluation  efforts  so  that  they  may  speak  with  a more  viable 
corporate  voice. 

This  study  report  examines  the  functions  and  capabilities  of  Naval  Material 
Command  (NAVMAT)  Test  and  Evaluation  Coordinators  to  provide  this  type  of  over- 
all assessment  and  concludes  that  no  administrative  support  system  yet  exists 
to  enable  the  coordinator  to  collect  and  display  corporate  test  and  evaluation 
data  in  usuable  report  formats. 

The  report  proposes  a computerized  test  and  evaluation  data  system  and  defines 
specific  input  and  report  output  formats  that  are  needed  by  the  organization 
to  assess  test  and  evaluation  status.  Ideas  for  future  system  sophistication  sure 
provided,  and  sample  report  formats  sure  included  in  the  appendix. 

Although  this  report  concentrates  on  Naval  Material  Command  requirements,  its 
test  and  evaluation  data  concepts  are  equally  applicable  to  other  Navy  organi- 
zations, other  Services  and  other  areas  of  endeavor  as  well. 
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EXECUTIVE  SUMMARY 


Changes  in  acquisition  policy  in  the'  19?0's  have  placed  increased 
emphasis  on  the  importamce  of  the  test  and  evaluation  (T&E)  process  and  on 
the  success  of  certain  T&E  milestones  eis  prerequisites  to  Defense  System 
Acquf  .-^ition  Review  Council  (DSARC)  decisions  and  the  commitment  of  further 
developnental  auid  production  funds.  Accordingly,  there  is  an  Increasing  need 
within  all  Department  of  Defense  (DoD)  communities  to  provide  more  visibil- 
ity to  critical  T&E  events  and  to  assess  their  significance  within  the  over- 
all process. 

Program  and  Acquisition  Managers  normally  coordinate  the  major  portion 
of  test  and  evaluation  efforts  within  the  framework  of  their  individual 
internal  organizations;  coordinating  unique  problems  auid  specific  require- 
ments with  higher  authority  on  am  as  occurring  or  as  required  basis.  In  this 
current  environment  of  ccmplex  acquisitions,  organizational  ccmimanders  now 
need  a more  integrated  auid  consistent  report  of  T&E  status  in  a corporate 
sense;  a means  of  assessing  the  total  T&E  posture  across  all  boundries  of 
the  organization.  Within  the  Naval  Material  Command,  Test  auid  Evaluation 
Coordinator  positions  have  been  established  to  provide  such  a service,  but 
no  awiministzative  support  system  has  yet  been  devised  to  enable  that  coor- 
dinator to  collect  and  easily  display  corporate  T&E  data  in  flexible  and  div- 
erse report  formats.  This  report  proposes  the  development  of  a ccmputer  data 
base  of  T&E  milestone  information  to  accomplish  this  purpose. 

There  is  an  inherent  human  resistance  to  the  use  of  management  informa- 
tion systems  (MIS)  as  a solution  to  problems  such  as  this.  One  reason  is  that 
the  MIS  is  not  capable  of  managing  anything,  but  is  often  sold  to  managers 
on  this  premise.  Consequently,  volumes  of  unusable  paper  are  generated  and 


discarded  daily  with  no  particular  function  being  performed. 

On  the  contrary,  the  information  system  proposed  herein  is  designed 
to  be  functional.  Its  purpose  is  to  collect  T&E  milestone  data,  primarily 
contained  in  Test  and  Evaluation  Master  Plauis  (TEHPs)  auid  other  program 
sources,  aind  reconstruct  the  data  into  useable  information  tools  for  senior 
mauiagers.  The  ccanputer  output  reports  described  in  this  paper  include  Box 
Scores  concerning  document  preparation.  Reports  of  Overdue  Milestones  and 
Major  Milestone  Listings  sorted  by  a number  of  grouping  options. 

Although  this  paper  is  oriented  toward  the  needs  of  the  Naval  Material 
Command  (NAVMAT)  and  Naval  Systems  Commands  (SYSCOMs) , the  test  and  evalua- 
tion reporting  procedures  discussed  are  ailso  applicable  to  T&E  organizations 
in  other  Navy  commands  and  in  other  Services  as  well.  In  addition,  the  con- 
cepts of  data  collection,  storaige,  flexible  sort-print  techniques  and  their 
utility  in  solving  numerous  coordination  problems  of  any  type  are  equally 
applicable  in  other  areas  of  endeavor.  This  paper  hopes  to  stimulate  thought 


toward  that  end. 
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1.0 


INTRODUCTION 


1.1  PURPOSE 

I can't  think  of  auiy thing  more  exciting  (other  than  walking  barefoot 
on  hot  rocks)  than  to  pick  up  a paper  entitled  " A statistical  Analysis 
of  the  Failure  Rate  of  Wirewound  Resistors  in  VHF  Receivers  Deployed  in 
Tropical  Satellite  Ground  Stations  " and  leaf  through  its  contents.  True, 
the  author  probably  devoted  six  months  of  intensive  research  effort, 
personally  derived  the  forty -five  pages  of  mathematical  formulas,  and  came 
to  the  honest  conclusion  that  half -watt  resistors  last  for  at  least  three 
years  if  you  don't  leave  the  receiver  out  in  the  rain  longer  than  2.3^ 
hours.  I grant  that  such  a paper  would  probably  be  considered  an  academic 
achievement  aind,  to  someone  designing  Braziliaui  satellites,  may  save  a 
significant  sunount  of  engineering  design  work.  But,  to  99^  of  us,  it  is  a 
paper  that  will  probably  be  read  once,  graded  by  a professor  and  placed  on 
a shelf  - never  to  be  seen  again  (except  for  those  who  know  the  author, 
because  he  ordered  2000  extra  copies  for  his  friends) . 

On  the  contrary,  my  purpose  is  to  write  a paper  which  is  functional; 
which  provides  a useful  product  that  can  be  applied  and  implemented.  In 
this  particular  case,  to  propose  a better  way  of  providing  visibility  to 
critical  test  and  evaluation  milestones  within  the  Naval  Material  Command. 
In  addition,  it  is  hoped  that  my  approach  will  provide  "food  for  thought" 
for  others  who  might  find  some  application  in  similar  situations. 


1.2  OBJECTIVES 


The  objectives  of  this  paper  are  as  follows i 

• To  describe  a requirement  within  the  Naval 
Material  Command  to  properly  coordinate  data 
concerning  test  and  evaluation  milestones. 

• To  briefly  discuss  the  motivating  DoD,  OPNAV 
and  Material  Commauid  test  aind  evaluation 
policies  and  environment  which  dictate  this 
requirement. 

• To  develop  the  framework  of  a data  collection 
and  Information  system  to  satisfy  this  need. 

It  is  also  importamt  to  mention  the  "non-objectives"  - those  areaa 
which  will  not  be  auldressed  due  to  existing  time  constraints! 

• This  paper  does  not  provide  a detailed  description  of  the 
evolution  of  test  auid  evaluation  in  the  Depairtment  of  Defense 
or  in  any  partlculatr  service.  The  reaider  is  referred  to  the 
Bibliography  where  a number  of  excellent  consolidations  axe 
listed  which  relate  the  findings  of  the  Blue  Ribbon  Defense 
Pauiel  in  1970  auid  the  subsequent  actions  which  evolved.  There 
will,  however,  be  a brief  discussion  of  these  policies  for  the 
purpose  of  providing  a baseline  knowledge  of  test  and  evaluation 
management  requirements. 

• Similarly,  this  paper  does  not  attempt  to  enter  the  arena  of 
the  pros  auid  cons  of  management  information  systems  in  general. 
Again,  the  Bibliography  lists  a number  of  documents  which 
specif icailly  treat  the  subject  as  it  applies  to  Project  Manage- 
ment Offices. 

• No  research  has  been  conducted  regarding  other  test  auid  evalua- 
tion data  beises  auid  manual/ computer  Information  systems  existing 
at  other  commands  or  within  other  Services.  There  are  many  excel- 
lent management  schemes  that  exist  at  commauids  such  as  Commander 


Operatloiuil  Teat  and  Evaluation  Force  (COMOPTEVFOR) . This  paper 
attempts  only  to  provide  a system  tailored  to  the  needs  of  a 
Naval  Systems  Command  Test  and  Evaluation  Coordinator. 

The  principles  discussed  in  this  paper  are  not  easily  applied 
to  some  program  eureas  such  as  the  ship  acquisition  process  per 
se  because  of  their  complexity  aind  uniqueness.  The  proposed 
approach  to  a test  and  evaJ-uation  information  system  may  not 
have  an  Immediate  application  in  these  cases;  however,  the 
concept  of  data  collection,  consolidation  and  control  is  still 
a basic  need  in  many  areas. 


1.3  SCOPE 

This  section  will  briefly  describe  the  contents  of  the  remainder  of 
the  paper  in  order  to  illustrate  the  continuity  of  presentation, 

1.3*1  Background 

In  Section  2.0  the  basic  DoD  policies  concerning  test  and 
evailuation  are  discussed,  including  a description  of  how  the  Navy  trans- 
lated these  policies,  through  Secretaury  of  the  Navy  (SECNAV)  and  Chief  of 
Naval  Operations  (CNO)  directives,  into  requirements  for  implementation 
hy  major  and  subordinate  commands.  Section  2.0  will  also  define  unique 
Acquisition  Categories  (ACATs)  which  determine,  in  accordance  with  dollar 
thresholds,  how  acquisition  projects  are  msmaged.  The  requirement  to  pre- 
pare suid  implement  a Test  and  EvaJ-uation  Master  Plan  (TEMP) , and  the 
requirement  to  obtain  an  Approval  for  Service  Use  (ASU)  for  a particular 
equipment  or  system  prior  to  obtaining  a release  of  production  funds,  are 
also  discussed. 

1.3*2  Management  of  T&E  Milestones  Within  the  Naval  Material  Command 
Section  3*0  discusses  Material  Ccaunand  T&E  directives  and 
briefly  describes  the  Systems  Command  (SYSCOM)  T&E  Coordinator  organization 
and  functions.  The  section  continues  with  a discussion  of  the  requiranent 
to  awiequately  document  and  coordinate  test  auid  evaluation  milestone  infor- 
mation on  the  System  Command  level  and  to  identify  current  T&E  Coordinator 
capabilities  to  provide  this  degree  of  detailed  management.  The  section 
concludes  that  additional  assistauice  is  required  auid  proposes  the  concept 
of  a T&E  information  data  base  as  a possible  solution. 

1.3*3  Development  of  an  Information  Systan 

Section  4,0  provides  a short  discussion  on  the  value  of  having 
an  information  system  (not  necessarily  a MANAGEMENT  information  system)  to 
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assist  in  performing  certain  functions,  and  relates  this  need  to  the  test 
and  evaluation  coordinator  responsibilities  previously  discussed. 

1,3.4  Development  of  a Test  and  Eivaluation  Information  System 

Section  $.0  and  5*1  continue  to  define  specific  types  of  test  and 
ev2d.uation  milestone  data  contained  in  TEMFs  euid  consolidates  this  data 
into  appropriate  input  categories  (Section  5*2)  for  use  in  a data  base 
matrix  from  which  variations  of  information  can  be  sorted,  extracted  amd 
printed  in  specific  report  formats.  Section  5*3  describes  these  output 
formats  which  include  monthly  box  scores  and  milestone  summauries  in 
numerous  categories. 

1.3*5  Summary 

Section  6.0  provides  a summation  of  the  report  auid  discusses 
potential  implementation  procedures  which  should  be  consider^.  The  section 
also  addresses  certain  thoughts  on  future  system  expansion  and  sophistica- 
tion. 

1.3*6  Appendix  amd  Bibliography 

The  Appendices  contain  supplsnentatry  information  on  the  contents 
of  TEMPs,  System  Commauid  T&E  Coordinator  functions,  a sample  Input  Data 
Worksheet  and,  most  Importamtly,  a computer  output  report  sample  of  each 
type  of  format  discussed  in  Section  5*3*  is  strongly  reccanmended  that 
the  reawier  briefly  scan  these  saunples  before  proceedirg  further  into  the 
report  - the  awivantages  of  having  reports  like  these  will  be  obvious. 

The  Bibliography  is  divided  into  topic  groups  in  order  to 
more  clearly  identify  certain  aureas  of  discussion  not  auldressed  in  detail 
in  this  paper.  A number  of  references  (Individual  Study  Projects)  aure 
included  since  these  efforts  "zero-in"  on  specific  aomais  of  interest 
as  opposed  to  sane  "general  philosophy"  type  of  references. 


2.0 


BACKGROUND 


2.1  GENERAL 

In  the  1960's,  SecretauTT  of  Defense  McNamara  directed  that  the 
acquisition  of  defense  systems  be  accomplished  by  using  a concept  of  Total 
Package  Procurement.  This  concept  involved  extensive  systems  analysis  and 
development  of  paperwork  approaches  which  were  used  as  the  basis  for  sub- 
sequent research,  development  and  production  (usually  by  a single  contrac- 
tor) . There  was  a general  concern  about  this  total  package  ptfiilosophy  since 
such  early  commitments  virtually  eliminated  any  flexibility  to  vary  ap)- 
proaches  auid  exercise  Innovative  thinking  along  the  path  from  concept  to 
operational  deployment. 

In  1970,  a Blue  Ribbon  Defense  Panel  concluded  that  a more  effective 
approach  to  systems  acquisition  would  be  to  set  achievable  goals  at  the 
onset  and  incrementally  develop  and  test  along  the  way  to  demonstrate 
achievement  of  these  goals.  The  Panel  focused  up)on  the  need  to  establish 
viable  test  and  evaluation  policies,  not  only  in  the  developmental  testing 
of  hardwaare  and  prototypes,  but  also  in  the  testing  of  operational,  per- 
formance prior  to  the  commitment  of  major  production  funds. 

This  background  section  will  identify  DoD  policy  guidance  that  was 
issued  as  a result  of  this  revaluation,  and  will  briefly  discuss  how  that 
policy  was  generally  interpreted  and  implemented  by  the  Secretary  of  the 
Navy  (SEGNAV)  and  the  Chief  of  Naval  Operations  (CNO) . As  indicated  earlier, 
this  paper  will  not  delve  into  the  historical  details  of  these  policies, 
but  will  lightly  expose  the  reader  to  the  baseline  requirements  which 
drive  the  need  to  properly  coordinate  test  auid  evaluation  milestones. 


6 


2.2  DEPARTMENT  OF  DEFENSE  (DOD)  POLICY 
2.2.1  DoD  Directive  5000.1 

Following  the  Blue  Ribbon  Defense  Panel's  recommendations, 
Secretary  of  Defense  Padrard  issued  DoD  Directive  5000.1  in  July  of  1971 
which  primarily  divided  the  acquisition  process  into  five  discernible 
phases;  identified  the  need  to  increnentally  review  the  progress  of  the 
program  at  specific  milestone  points  (by  a body  known  aa  the  Defense  System 
Acquisition  Review  Council  (DSARC) ) , and  meJce  decisions  whether  or  not  to 
continue  the  effort  and  ccanmit  additional  developtientail  and  production 
funds.  The  directive  further  identified  test  and  evaJ.uation  as  one  key 
means  of  providing  supporting  Inputs  to  the  prescribed  review  process. 

This  guidance  was  fvirther  revised  by  Secretary  of  Defense 
Clements  by  reissuing  DoD  Directive  5000.1  on  18  January  1977,  which  div- 
ided the  acquisition  process  into  the  following  sequence  of  key  milestone 
decision  points  and  i^iasesi 

• MILESTONE  ^ (Program  Initiation) 

- Conceptual  Phase 

• MILESTONE  I 

- Demonstration  and  Validation  Phase 

• MILESTONE  II 

- Full  Scale  Engineering  Developnent  Phase 

• MILESTONE  III 

- Production  and  Deployment  Phase 

The  acquisition  process  is  initiated  with  the  approval  of  a Mission  Element 
Needs  Statement  (MENS)  and  continues  through  a sequence  of  decision  events 
(Milestones  and  DSARC  reviews)  designed  to  evaluate  and  recognize  the 
achievement  of  established  program  objectives. 
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DoD  Directive  5000.1  does  not  specifically  spell  out  the  me- 
chanics of  how  test  and  evaluation  will  be  Implemented  within  these  phases, 
tut  instead  provides  the  following  general  policy  (li8)^i 

" U.  Test  and  evaluation  shall  commence  as  early 
as  possible.  An  estimate  of  military  utility 
5Uid  of  operational  effectiveness  and  oper- 
ational suitability  including  logistic  support 
requirements,  shall  be  made  prior  to  large- 
scale  production  canmitments.  The  most  real- 
istic test  environment  possible  and  an  accept- 
able representation  of  the  future  operational 
systan  will  be  used  in  the  testing.  " 

2.2.2  DoD  Directive  5000.2 

Whereas  DoD  Directive  5000.1  establishes  the  overall  policies 
for  the  acquisition  process,  DoD  Directive  5000.2  (also  issued  in  January 
1977)  provides  procedures  to  implement  this  guidance,  establishes  review 
councils,  assigns  responsibilities  and  further  defines  the  milestone 
decision  points  amd  acquisition  phases  by  describing  specific  detailed 
actions  to  be  accomplished.  Test  and  evaluation  is  not  referred  to  as  a 
separate  action.  More  importantly  it  is  treated  as  a "given";  an  assump- 
tion which  permeates  each  of  the  other  actions  such  as  stating  needs  for 
"demonstrations  at  the  system  level",  references  to  "supporting  OT&E" 
and  the  like.  A significant  single  recognition  of  test  and  evaluation, 
however,  is  reflected  by  including  a T&E  Advisor  to  the  DSARC  with  the 
following  responsibility  (2iEncl  1)  i 

" The  Deputy  DDR&E  (T&E)  shall  participate  in 
DSARC  reviews  and  shall  report  to  the  DSARC 
and  to  the  Secretary  of  Defense  on  test  plaui- 
nlng  and  results” 


The  notation  (#i^)  is  used  in  this  report  to  identify  the  applicable 
reference  for  the  fact  or  quote  stated.  The  first  digit  is  the  number  of 
the  reference  ais  listed  in  the  bibliography  appearing  at  the  end  of  the 
paper.  The  second  digit  identifies  the  page  number  within  that  reference. 
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2.2.3  DoD  Directive  5000.3 


This  directive,  entitled  "Test  aM  Evaluation",  is  the  key 
policy  document  which  governs  the  implementation  of  T&E  throu^out  all 
DoD  Components,  and  is  the  basis  for  the  Navy  T&E  directives  and  guidance 
discussed  in  subsequent  sections.  The  directive  defines  certain  types  of 
T&E  to  be  performed  during  the  acquisition  cycle  as  follows! 

• Development  Test  and  Evalviation  (DT&E) 

DT&E  is  that  test  and  evaluation  conducted  toi 

- Demonstrate  that  the  engineering 
design  and  development  process 
is  complete. 

- Demonstrate  that  the  design  risks 
have  been  minimized. 

- Demonstrate  that  the  system  will 
meet  specifications. 

- Estimate  the  system's  military 
utility  when  introduced. 

• Operational  Test  and  Evaluation  (OT&E) 

OT&E  is  that  test  and  evaluation  conducted  toi 

- Estimate  the  prospective  system's 

• Military  Utility 

• Operational  Effectiveness 

• Operational  Suitability 

Including  compatibility,  inter- 
operability, reliability,  main- 
tainability, and  logistic  train- 
ing requirements. 

- Provide  information  on  organization, 
personnel  requirements,  doctrine  and 
tactics . 

- Provide  data  to  support  or  verify 
material  in  operating  instructions, 
publications  and  hauidbooks. 

• Production  Acceptance  Test  amd  Evaluation  (PAT&E) 

PAT&E  is  test  and  evaluation  of  production 

items  toi 

- Demonstrate  that  the  items  procured 
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fulfill  the  requirements  eind  specifi- 
cations of  the  procuring  contract  or 
agreements • 


In  addition  to  the  three  major  test  auid  evaluation  categories 
listed  above,  DoD  Directive  5000.3  also  provides  fori 


• Test  and  Bvalxiation  for  Major  Ships  of  a Class 

• Test  and  Evaluation  for  One-of-a-Kind  Systems 


For  the  purpose  of  this  paper,  the  directive  provides  the  T&E 
policy  baseline  and  justifies  the  importance  of  the  results  of  test  and 

evaluation  in  the  decision  process  and  the  necessity  to  properly  plan  and 

; 

coordinate  such  testing  through  official  dociimentatlon . First,  the  direc- 
tive emphasizes  that  (3«2)  i 


" Acquisition  schedules  will  be  based,  inter 
ailla,  upon  accomplishing  test  auid  evaluation 
milestones  prior  to  the  time  that  key  decisions 
which  would  ccanmit  significant  added  resources 
acre  to  be  made." 


Secondly,  it  directs  that  a Test  and  Evaluation  Master  Plan  (TEMP)  be 
developed  for  each  program  ais  follows  (3*6)  i 

" The  DoD  Gcanponent  will  prepare  as  early  as 
possible  in  the  acquisition  process,  and 
prior  to  initiation  of  Full-Scale  Develop- 
ment, an  overall  test  and  evaluation  plan  to 
identify  and  Integrate  the  effort  auid  sched- 
ules of  all  T&E  to  be  accomplished  and  to 
Insure  that  all  necessary  T&E  is  accomplished 
prior  to  key  decision  points.  The  TEMP  will 
be  kept  current  hy  the  DoD  Gcanponent." 


Accordingly,  this  paper  will  propose  a structured  information 
system  which  consolidates  the  milestone  dates  nuid  key  planning  data  that 
is  contained  in  this  TEMP.  At  this  moment,  DoD  Directive  5000.3  is  being 
revised  as  a result  of  changes  in  the  acquisition  process  brought  on  by 
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the  IssiJance  of  new  DoD  Directives  5000.1  and  5000.2  in  January  1977.  It 
is  understood  that  the  proposed  new  5000.3  will  provide  greater  emphasis 
and  visibility  to  the  TEMP  and  the  importance  of  its  planning  and  imple- 
mentation role.  The  contents  of  the  TEMP  and,  more  specifically,  the  T&E 
milestones  to  be  tracked,  will  be  discussed  in  more  detail  in  Sections 
3.4  and  5*2  . 
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2.3  SECRETARY  OF  THE  NAVY  (SECNAV)  POLICY 
2.3.1  SECNAVINST  5000.1 

At  the  next  lower  echelon  within  DoD,  SECNAVINST  5000.1 
reflects  the  SECNAV  policy  in  response  to  DoD  Directives  5000.1  and  5000.2. 
Again,  this  is  general  policy  with  the  details  left  to  the  Chief  of  Naval 
Operations  (CNO)  to  develop  and  implement  (discussed  in  Section  2.4). 
Basically,  the  SECNAV  Instruction  recognizes  that  (4il3)  « 


" The  wide  variety  of  naval  weapons  dictates 
varying  approaches  to  the  conduct  of  test 
and  evaluation;  such  effort  shall  be  tailored 
to  the  needs  eind  characterizations  of  each 
individual  acquisition  — prime  consideration 
being  given  to  aidequate  operationally  oriented 
testing" 

The  directive  further  establishes  a general  sequence  of  events  within  the 
Navy  T&E  process,  discusses  the  importance  of  operational  T&E  (OT&E),  and 
applies  T&E  policy  to  new  acquisitions  including  those  not  subject  to  the 
thresholds  of  DoD  Directive  5000.1. 

Although  SECNAVINST  5000.1  does  not  specifically  discuss  the 
TEMP  or  required  Navy  T&E  milestones  to  be  Included,  the  concept  of  proper 
milestone  planning  eind  coordination  throughout  the  process  is  reflected 
by  the  following  statement  (4tl4)  t 

" Test  aJid  evalioatlon  effort  shall  be  effectively 
correlated  with  previously  outlined  requirements 
concerning  approval  of  material  for  service  use. 

Specif IcaJ-ly,  the  procedural  aspects  of  require- 
ments determination,  research  and  development, 
manufacture  of  service  test  model(s),  technical 
evailiiation,  initial  operational  test  auid  evalua- 
tion, full  operational  evaluation  auid  approval 
for  service  use  shall  be  correlated.  " 
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2.4  GHIEy  OF  NAVAL  OPERATIONS  (C3^0)  ACQUISITION  POLICY 


2.4.1  OPNAVINST  5000 .42A 

In  the  next  echelon  of  Navy  canmand,  the  broaxi  SECNAV  policy 
above  is  further  defined  and  implemented  in  OPNAVINST  5000. 42A  of  3 March 
1976.  The  directive  basically  identifies  Navy  acquisition  meuiagement 
policies,  defines  the  required  documentation  (such  as  Navy  Decision  Coor- 
dinating Papers  (NDCPs) , Operational  Requirements  (ORs) , etc.),  establishes 
review  committees  and  panels,  and  provides  specific  outlines  and  formats 
for  major  documentation. 

OPNAVINST  5000. 42A  awidresses  two  significant  factors  which 
directly  impact  upon  the  concept  and  implementation  of  test  and  evaluation 
procedures  in  the  Navy.  These  factors  axei 

• The  establishment  of  Acquisition  Categories  (ACATs) , and 

• The  reinforced  requirement  to  prepare  and  prosecute 

a Test  and  Evaluation  Master  Plan  (TEMP). 

2.4.2  Acquisition  Categories  (ACATs) 

Navy  acquisition  programs  aure  divided  up  into  four  categories 
in  accordance  with  specific  dollair  value  thresholds.  These  categories 
govern  the  method  and  type  of  acquisition  procedures  to  be  applied,  aind 
determine  respective  decision  authority  levels  for  each.  The  following 
partiaul  criteria  aire  assigned  for  each  category* 

• ACAT  I 

- Major  programs  having  an  estimated* 

• RDT&E  cost  in  excess  of  $75  million,  or 

• Production  cost  in  excess  of  $300  million 

- Decision  authority  is  SEGDEF/DEPSEGDEF 

- Decision  Coordinating  Paper  (DCP)  is 
normailly  required 


• AGAT  II 


- Other  programs  having  an  estimated! 

• RDT&E  cost  in  excess  of  $20  million,  or 

• Production  cost  in  excess  of  $50  million 

- Decision  authority  is  SEC2NAV 

- Program  Memorandum  is  normally  required. 

• AGAT  III 

- Programs  below  the  AGAT  II  level  having 
an  estimated: 

• RDT&E  cost  in  excess  of  $5  million,  or 

• Production  cost  in  excess  of  $20  million 

- Decision  authority  is  the  PROGRAM  SPONSER 

- Navy  Decision  Goordinating  Paper  (NDGP) 
is  normally  required 

• AGAT  IV 

- Programs  not  in  AGATs  I,  II  or  III  having: 

• RDT&E  cost  less  than  $5  million,  or 

• Production  cost  less  than  $20  million 

- Navy  Decision  Goordinating  Paper  (NDGP) 
is  normally  required 

In  addition  to  the  criteria  above,  programs  below  the  AGAT  III 
(i.e.  AGAT  IV ) dollar  thresholds  may  be  designated  as  AGAT  III  programs 
if  they  directly  affect  the  military  chaxacterlstics  of  ships,  alrcratft,  or 
other  combatauit  units;  or  if  they  require  operational  testing  to  support 
key  program  decisions,  or  require  fleet  RDT&E  support.  A knowledge  of  these 
acquisition  categories  is  also  important  in  vinderstanding  test  and  evalua- 
tion policy  since  they  govern  whether  or  not  a TEMP  is  required. 

2.4.3  TEMP  Requirement 

OPNAVINST  5000. 42A  empheisizes  that  the  Test  and  Evaluation 
Master  Plan  (TEMP)  is  the  controlling  management  document  which  defines 
test  auid  evaluation  for  each  navy  acquisition  program  amd  indicates  that 
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it  is  to  be  prepared  in  accordance  with  the  primary  implementing  document 
OPNAVINST  3960.10.  Section  2.5*2  discusses  the  structure  of  the  TEMP  in 
more  detail. 


2.5  CHIEF  OF  NAVAL  OPERATIONS  (CNO)  TEST  AND  EVALUATION  POLICY 


2.5.1  OPNAVINST  3960.10 

This  instruction  is  the  primairy  Navy  document  for  implementing 
the  DoD  test  and  evaluation  policies  set  forth  in  DoD  Directive  5000.3. 

In  addition,  the  instruction 1 

• Defines  the  T&E  responsibilities  for  the 
major  participants 

• Establishes  internal  ixrocedures  for  the 
plauining,  conducting  and  reporting  of 
T&E  efforts,  and 

• Delineates  the  complimentatry  relationship 
existing  between  DT&E  and  OT&E  during  the 
life  of  an  acquisition  program. 

But  most  importantly  as  far  as  this  study  project  is  concerned,  OPNAVINST 
3960.10  describes  in  detail  the  requirements  for  the  preparation  of  two 
significant  T&E  documents! 

• A Test  and  Evaluation  Master  Plsui  (TEMP) 

- Discussed  in  Section  2.5.2 

• Certification  of  Readiness  for  OPEVAL 

- Discussed  in  Section  2.5.3 

2.5.2  Test  suid  Evaluation  Master  Plan  (TEMP) 

2.5.2. 1 General  Description 

The  TEMP  is  a short,  concise  master  planning  document 
which  describes  the  test  and  evaluation  effort  for  a particular  ACAT  I,  II 
or  III  program.  Its  specific  purposes,  as  stated  in  OPNAVINST  396O.IO 
(6iEncl  3)  are  i 

• To  direct  auid  control  the  accomplishment 
of  auiequate  T&E 

• To  identify  all  required  T&E  resources 

• To  facilitate  long-range  planning, 
programming  and  budgeting 
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• To  eliminate  redundant  testing,  and 

• To  reduce  Fleet  RDT&E  Support  requirements 
to  the  essential  minimum 

The  TEMP  is  the  controlling  T&E  management  document 
and,  as  such,  contains  the  total  integration  of  requirements  for  both  the 
Developing  Agency  (DA)  for  DT&E,  and  for  COMOPTEVFOR  for  OT&E,  and  the 
integration  of  schedule  and  resources  required  for  acccoiplishment.  Cost 
and  time  implications  are  clearly  spelled  out  to  permit  reviewers  to  assess 
the  resources  being  committed  to  test  and  evaluation. 

2. 5.2. 2 Format 

The  TEMP  is  limited  to  20  pages  or  less  in  length  and 
contains  sufficient  detail  on  how  and  when  identified  or  suspected  uncer- 
tainties will  be  resolved.  In  order  to  provide  consistency  in  the  type  of 
information  and  level  of  detail  provided,  aill  TEMPs  are  prepared  in  accord- 
ance with  the  following  general  outline! 


PART  I. 

.ADMINISTRATIVE  INFORMATION 

PART  II. 

DESCRIPTION 

PART  III. 

INTEGRATED  SCHEDULE 

PART  IV. 

DT&E  OUTLINE 

PART  V. 

OT&E  OUTLINE 

PART  VI. 

PAT&E  OUTLINE  (if  Applicable) 

PART  VII. 

RESOURCE  SUMMARY 

PART  VIII. 

REFERENCES 

A further  breakdown  of  this  outline,  illustrating  the  next  level  of  detail- 
ed information  to  be  included,  is  provided  in  Appendix  A of  this  report. 

Each  TEMP  is  assigned  a number  which  is  the  same  as 
the  T&E  Identification  Number  (TEIN)  assigned  by  GNO  (OP-983)  auid  as  listed 
in  a quarterly  document  entitled  "GNO  Index  of  Acquisition  Programs"  ^lO). 
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2.5.2. 3 Temp  Preparation  and  Sutmission 

The  detailed  TEMP  preparation,  sutmission  criteria 
and  procedures  are  beyond  the  scope  of  this  report  auid  the  reader  should 
refer  to  OPNAVINST  3960.10  auid  local  command  implementing  directives  for 
further  details.  However,  the  following  specific  items  are  cited  in  order 
to  provide  a better  appreciation  of  the  purpose  of  the  TEMP  (6i7&8)  i 

• CNO-approved  TEMPs  are  required  for  all  AGAT  I,  II 
and  III  programs.  For  AGAT  IV  programs,  the  Ghief 
of  Naval  Material  (GHNAVMAT)  has  promulgated  instr- 
uctions for  the  preparation  auid  approval  of  Test  aM 
EJvaluation  Plans  (TEIPs) , as  discussed  in  Section 

3. 2. 1.2. 

• TEMPs  are  prepared  hy  the  Developing  Agency  (DA) 
in  coordination  with  other  DA  codes  (for  internal 
review) ; with  GOMOPTEWOR  (for  finalizing  the 
OT&E  Outline) ; and  with  NAVMAT  (for  review  and 
comments) . A smooth  TEMP  is  then  prepared  auid 
forwaurded  to  GNO. 

• The  approval  of  the  TEMP  constitutes  GNO  direction 
to  conduct  the  T&E  program  defined  therein.  Result- 
ant test  plans  will  be  consistent  with  TEMP  pro- 
visions. 

• The  TEMP  will  be  reviewed  and  updated  at  least 
auinually  and  about  two  months  prior  to  major 
decision  milestones  (DSARC  or  equivllent)  to  in- 
corporate slgnlf  icamt  results  achieved  auid  chauiges 
to  plans  and  milestones. 

2. 5 .2. 4 TEMP  Application  to  Report  Objective 

A brief  description  of  the  NAVMAT/SYSGOM  T&E  organi- 
zation will  be  provided  in  Section  3*0.  A requirement  will  be  expressed 
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within  that  framework  to  properly  identify,  make  visible  and  coordinate 
information  concerning  T&E  milestone  data  contained  in  PART  III  of  the  TEMP 
(integrated  Schedule).  Section  5*0  will  propose  the  development  of  an  in- 
formation system  with  an  input  data  format  consistent  with  the  structure 
of  milestone  information  in  the  TEMP. 

2.5.3  Certification  of  Residlness  for  OPEVAL 

Before  a system  or  equipment  can  be  tested  in  the  final  phase 
of  OT&E  (OT-IIIb  "OPEVAL"),  prior  to  a production  decision  at  DSARC  III,  it 
must  be  certified  ty  the  Developing  Agency  that  is  has  successfully  complet- 
ed the  final  phaise  of  DT&E  (DT-IIIb  "TECHEVAL")  and  that  it  is  "ready  for 
OPEVAL".  This  certification  is  sulanltted  to  GNO  in  the  form  of  a report 
addressing  full  compliance  to  certification  criteria,  or  requesting  waivers 
for  minor  items.  A complete  listing  of  such  criteria  is  found  in  Enclosure 
(2)  of  OPNAVINST  3960.10,  auid  are  summarized  as  follows  1 

• DT&E  objectives  have  been  met 

• High  probability  of  successful 
OPEVAL  performance  exists 

• Adequate  logistics  support  has 
been  provided  (plans,  spare  parts, 
manuals , etc) 

• Personnel  manning  approximates 
operating  conditions 

• Reparesentative  training  has  been 
conducted 

• System  configuration  is  equivalent 
to  the  intended  production  system 

The  Certification  of  Readiness  for  OPEVAL  is  a significant  test 
2ind  evaluation  milestone  for  the  Developing  Agency  and,  as  such,  will  be 
considered  as  a key  data  point  in  the  test  and  evaluation  information  sys- 
tem being  proposed  in  this  paper. 
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2.6  APPROVAL  FOR  SERVICE  USE  (ASU) 


In  addition  to  the  TEMP,  which  provides  aui  integrated  schedule  of  T&E 
milestones,  and  the  requirement  for  Certification  of  Readiness  for  OPEVAL, 
all  systems  or  equipments  developed  hy  the  Navy  and/or  which  the  Navy  in- 
tends to  support  must  be  "approved  for  service  use"  prior  to  commitment  to 
major  production.  This  requirement  is  directed  by  OPNAVINST  4720. 9D,  and 
certain  provisions  of  that  instruction  are  provided  in  the  following  sub- 
sections . 


2.6.1  Full  Approved,  for  Service  Use  (ASU) 

Commonly  refeired  to  as  just  "ASU",  this  approval  is  the  total 
and  complete  approval  required  in  order  to  release  production  funds.  The 
requirement  is  defined  as  follows  (7i6)  i 


" No  new  system  or  significant  alteration  to 
an  existing  system  shall  be  approved  for 
production  until  it  has  been  adequately 
tested,  proven  operationally  suitable,  and 
determined  to  be  logistlcally  supportable." 


2.6.2  Provisional  APTaroval  for  Service  Use  (PASU) 

If  a particul2ir  system  or  equipment  is  to  be  produced  in  lim- 
ited quantity  initially,  as  a result  of  a DSARC  II  decision  or  on  the  basis 
of  a CNO  Evaluation  Boaurd  (CEB)  decision,  then  the  following  provision  of 
OPNAVINST  4720.90  applies  (7i5)  « 

" A 'provisional  approval'  may  be  sought  on 
programs  for  which  sufficient  operational 
testing  to  support  a final  determination  of 
approval  for  service  use  cannot  practically 
be  accomplished  prior  to  making  an  initial 
production  commitment." 

2.6.3  ASU  Application  to  ReT)ort  Objective 

Again,  the  detailed  procedures  suid  implications  of  this  re- 
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quirement  axe  not  within  the  scope  of  this  report;  however,  approval  for 
service  use,  like  certification  of  reeidiness  for  OFEVAL,  is  another  sig- 
nificauit  key  TisE  milestone  that  will  be  addressed  in  the  T&E  information 
system  proposed  in  Section  3*0,  and  the  brief  discussion  above  was  included 
in  order  to  familiarize  the  reader  with  this  requirement. 
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3.0  MANAGEMENT  OF  T&E  MILESTONES  WITHIN  THE  NAVAL  MATERIAL  COMMAND 

3.1  GENERAL 

The  preceding  section  outlined  the  test  and  evaluation  policy  hierarchy 
from  the  Department  of  Defense,  through  the  Secretaury  of  the  Navy,  to  the 
Chief  of  Naval  Operations  - the  point  where  iDasic  policy  was  translated 
into  specific  implementing  instructions  for  the  Navy  establishment.  These 
instructions  included  the  interpretation  of  policy  as  applies  to  unique 
Navy  sitiiations  and  environments,  service-peculiar  definitions,  documenta- 
tion and  procedures,  and  particularly  the  introduction  of  three  significant 
requirements  t 

• Test  sutid  Evaluation  Master  Plan  (TEMP) 

• Certification  of  Readiness  for  OPEVAL 

• Approval  for  Service  Use  (ASU) 

This  section  will  describe  the  general  T&E  organization  of  the  Naval 
Material  and  Systems  Commands,  and  the  documentation  developed  to  respond 
to  the  implementation  direction  of  the  CNO.  Again,  the  Intent  of  this  sec- 
tion is  not  to  cover  these  organizational  aspects  in  great  detail.  Its 
only  purpose  is  to  acquaint  the  reader  with  the  environment  of  NAVMAT  T&E 
coordination,  its  resources,  current  capabilities  and  the  continual  and  in- 
creasing need  for  detailed  T&E  Information  by  high  level  msuiagement  i)erson- 
nel. 
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3.2  NAVAL  MATERIAL  COMMAND  (NAVMAT) 
3.2.1  NAVMAT  Documentation 


3. 2. 1.1  NAVMATINST  3960. 6a  - Test  and  Evalxiation 

This  Instruction  Implements  OPNAV  T&E  policy  within 
the  Naval  Material  Commsuid  aind  establishes  policy  guidelines  for  T&E  and  for 
Acquisition  Category  TV  (ACAT  IV)  programs.  Specifically,  the  instruction  1 

• Identifies  the  responsibility  to 

- Review  program  formats 

- Review  T&E  documentation  and  plans 

- Insure  the  euiequacy  of  plans  for  the 
support  and  formeJ.  certification  of 
readiness  for  OPEVAL 

• Outlines  specific  procedures  for  the 
preparation,  processing  and  submission 
of  TEMPs  (and  "TEPs"  discussed  below) 

• Directs  that  all  System  Ccmunand  (SYSCOM) 

T&E  Focal  Points  (Coordinators)  maintain 
copies  of  all  TEMPs  and  TEPs  under  their 
cognizance  (a  repository) 

• Establishes  detailed  procedures  for  ACAT  IV 
T&E  pleuis  and  program  reviews 

• Requires  the  submission  of  a Test  and 
Evaluation  Plaui  (TEP)  for  projects  below 
the  ACAT  III  threshold  which  havei 

- A unit  cost  of  more  than  $10,000 

- A total  project  cost  of  $1  million 

- Hardware  that  requires  formal  ASU 

3. 2. 1.2  Test  and  Evaluation  Plan  (TEP) 

The  Test  and  Evaluation  Plan  (TEP)  is  the  management 
docxunent  which  describes  and  integrates  the  test  and  evaluation  effort 
auid  overall  T&E  schedule  for  a program  in  ACAT  IV.  Its  purpose  is  to  direct 
and  control  the  accomplishment  of  eulequate  T&E,  identify  required  resources 
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and  provide  increased  T&E  visibility  for  lower  cost  programs.  The  TEP  is 
similar  in  format  to  the  TEMP  except  there  is  no  operational  testing  (no 
OT&E  is  required  for  AGAT  TV  programs)  and  therefore  no  OT&E  outline  re- 
quired. Other  selected  elements,  depending  on  the  nature  of  the  program, 
may  be  excluded  from  the  TEP  as  long  as  the  basic  purpose  of  the  plan  is 
not  diluted.  The  TEP  is  to  be  20  pages  or  less  and  is  comprised  of  seven 
parts  (Appendix  A TEMP  Outline  less  PART  V ) . 

3. 2. 1.3  NAVMATINST  4720.1  - Approval  for  Service  Use  (ASU) 

The  basic  principles  for  approval  for  service  use  were 
discussed  in  Section  2.6.  The  GNO  has  delegated  the  authority  for  approval 
for  service  use  (ASU)  to  the  Ghlef  of  Naval  Material  (GHNAVMAT)  for  those 
equipments  suid  canponents  which  fall  below  the  "less  than  major"  dollax 
value  thresholds  (essentially  less  than  AGAT  l)  (9i2).  The  NAVMATINST 
4720.1  series  instructions  implement  the  OPNAV  policy  in  this  aorea  and  pro- 
vide detailed  guidance  to  material  ccanraands  including  the  delineation  of 
approval  authority,  prerequislties  and  procedural  actions.  The  ASU  milestone 
is  an  extremely  important  date  to  be  closely  manaiged  and  visibly  documented 
by  NAVMAT  and  SYSGOM  T&E  Goordlnators . 

3«2.2  NAVMAT  T&E  Organization  and  Functions 

It  is  difficult  to  "bound"  areas  of  functional  responsibilities 
because  of  the  chain  of  command  concept.  That  is,  even  though  a particular 
organization  code  (office)  may  be  charged  with  an  overall  function  (like 
T&E  Goordlnation) , there  may  be  only  one  or  two  individuals  chstrged  with 
a specific  function  within  that  category.  A case  in  point  is  the  T&E  Goor- 
dlnator,  or  more  specifically  the  individual  assigned  the  responsibility  for 
the  coordination  of  T&E  milestones  and  the  management/processing  of  TEMPs 


and  TEPs,  It  would  be  erroneous  to  attribute  the  function  to  the  laa:ger 
number  of  personnel  within  the  office  itself. 

For  example,  although  a number  of  responsibilities  dealing 
with,  and  impacting  with,  test  and  evaluation  efforts  within. the  Naval 
Materietl  Command  reside  within  Code  MAT  08  on  the  NAVMAT  HQ  staff,  only 
one  indlvidiULl  (NAVMAT  Code  08E12)  Is  involved  in  the  daily  mechanics  of 
TEMP/TEP  coordination  and  overseeing  the  numerous  SYSCOM  T&E  functions 
discussed  in  Section  3* 3* 3*  He  is  responsible  for  insuring  that  T&E  doc- 
umentation (TEMPs.  TEPs,  requests  for  ASU  and  numerous  other  correspondence) 
is  subnit  ted  in  a timely  msumer,  that  they  contain  the  required  aind  correct 
data,  and  that  the  documentation  has  been  properly  coordinated  with  aind 
subnitted  to  exterml  activities. 

The  NAVMAT  T&E  Coordinator  is  additionally  responsible  for  main- 
taining up-to-date  status  on  all  such  documentation  and  status  on  all  sig- 
nificant T&E  actions  within  the  Naval  Material  Command  of  interest  to 
Senior  Management  officials.  This  encanpassing  responsibility,  essentially 
resting  on  single  individuals  (including  SYSCOMs  below) , is  the  major  moti- 
vating reason  for  proposing  the  data  information  system  in  Section  5*0  as 
a mamagement  tool  to  assist  these  individuals  in  keeping  track  of  the  multi- 
tudinous data  involved.  Hence,  the  purpose  of  discussing  NAVMAT  and  SYSCOM 
T&E  Coordinator  functions  in  this  section  is  to  provide  a measure  of  justi- 
fication for  this  paper. 
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3.3  SYSTEMS  COMMANDS  (SYSCOMS) 

3.3*1  SYSCOM  Documentation 

Test  and  evaluation  documentation  on  a systems  command  level  is 
basically  a re-emphasis  of  the  rather  explicit  OPNAV  policy  (OPNAVINST 
3960.10),  as  modified  by  certain  unique  requirements  (such  as  TEP  prepara- 
tions). SYSCOM  policy  directives  apply  these  requirements  to  the  particu- 
^ lax  command  environment  and  tailor  the  procedural  aspects  to  Individual 

command  structures.  SYSCOM  T&E  directives  provide  detailed  instructions  on 
documentation  preparation,  review  suid  submission . 

[ 3.3.2  SYSCOM  T&E  Organization 

In  addition  to  SYSCOM  T&E  policy  documentation,  each  commajid 
has  separately  issued  an  instruction  or  notice  establishing  a T&E  Coordi- 
nator responsible  for  properly  administering  required  T&E  functions,  and 
designating  him  to  act  as  the  command  focal  point  for  processing  TEMPs,  TEPs, 
approval,  for  service  use  requests,  and  so  forth.  The  T&E  organizational 
structure  among  the  SYSCOMs  is  varied.  Without  going  into  great  detail, 
the  compaurlsons  below  are  provided  only  to  Illustrate  the  relative  capa- 
bilities that  exist. 

One  SYSCOM  has  established  the  T&E  Coordinator  position  as  a 
statff  code  and  hats  assigned  one  individual  to  be  responsible  for  all  ccxnmand 
TEMP/TEP  processing,  status  keeping  and  problem  solving  during  T&E-type 
crisis  periods.  Another  SYSCOM,  because  of  a larger  scope  of  T&E  efforts 
Involved,  hais  established  aui  office  with  six  or  seven  people  assigned.  Not 
aJ.1,  however  (ais  pointed  out  earlier),  axe  involved  with  the  TEMP/TEP/ASU 
functions  adone.  A third  SYSCOM  has  established  a highly  orgauiized  test  auid 

I 

evaLLviatlon  structure,  headed  by  a Flaig  Officer,  amd  composed  of  a number  of 
clearly  defined  functions  auid  divisions.  Heavy  reliauice  is  placed  on  the 
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project  management  structure  to  respond  to  the  T&E  details  of  their  indi- 
vidual programs.  Although  the  requirement  to  collect  and  coordinate  T&E 
milestone  data  still  exists  on  the  systems  canmand  level,  this  organization 
cannot  be  compared  to  the  "singulaurly  staffed"  coordinator  functions  of  the 
other  SYSCOMs  and  NAVMAT  Staff. 

3.3*3  SYSGOM  T&E  Coordinator  Functions 

Like  the  mix  of  organizationeil  structures  above,  the  functions 
of  SYSGOM  T&E  Coordinators  are  not  all  standard  since  real-world  functions 
are  tailored  to  the  individual  command  requirements.  However,  certain 
basic  functions  aure  essentially  common  among  the  single  coordinator  posi- 
tions. Each  is  basically  responsible  for  those  command  test  and  evaluation 
policies  and  practices  that  ha’.  e a general  applicability  to  all  SYSGOM  Prog- 
rams. They  axe  ailso  responsible  for  the  effective  utilization  of  SYSGOM 
T&E  ansets  and  facilities  (where  applicable) , and  serve  as  the  central 
point  of  contact  for  all  applicable  T&E  matters.  Appendix  B provides  a 
detailed  listing  of  specific  SYSGOM  T&E  Coordinator  functions.  They  can  be 
generally  categorized  as  follows i 

• Develops  and  Maintains  Command  T&E  Policies 

• Monitors  T&E  Status  and  Provides  Status  Reports 

• Provides  Assistance  to  Project  Managers  and 
Acquisition  Managers  in  the  Prepaxation  of 
Documentation 

• Reviews  and  Processes  TEMPs  and  TEPs 

• Reviews  and  Processes  Requests  for  Approvail 
for  Service  Use 

• Administers  Command  T&E  Support  Funds 

• Coordinates  T&E  Matters  with  External  Activities 

• Coordinates  Specifically  Assigned  T&E  Programs 
and  Projects 

• Manaiges  Command  T&E  Training  and  Education 
Programs 
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It  should  be  emphasized  that  these  SYSCOM  T&E  Coordinator 
functions  are  prlmaxily  a "staff-type"  duty  within  the  organization  - a 
focal  point  of  T&E  expertise  and  awimlnistrator  of  general  command  T&E  policy. 
The  next  working  layer  of  management  exists  in  the  Project  Manager/Acquisl- 
tlon  Manager  orgauilzations  themselves,  where  the  "nitty-gritty"  details  of 
test  and  evaluation  are  plsinned,  coordinated  emd  executed. 


3.4  CONTROL  OF  T&E  TEWP/TEP  MILESTONE  DATA 


3.4.1  The  Requirement 

The  Program/Project  Manager  (PM)  organization  normally  has  at 
least  one  position  designated  as  a T&E  manager.  If  that  function  is  not  being 
implemented  or  monitored  by  a full-time  designated  individual,  at  least 
he  will  appoint  someone  to  perform  additional  duty  in  that  area.  If  the 
PM  organization  is  small  and/or  personnel  resources  are  sparse,  and  work- 
load prevents  the  assignment  of  a specific  T&E  function,  the  effort  is 
performed  ais  an  integral  part  of  each  divisional  task  and  overall  T&E 
status  is  monitored  by  the  PM  himself.  The  message  is  that,  no  matter  how 
it  is  performed,  the  T&E  function  is  never  deferred  totally.  Although  the 
proper  amount  of  testing  "eats"  money  and  is  a planning  and  schedule  head- 
ache, it  is  recognized  that  it  is  a directed  effort  and  hi^ily  visible 
within  DoD.  Successful  T&E  is  one  key  to  a successful  DSARC. 

Accordingly,  the  Project  Manager  is  normally  up  to  speed  at 
auiy  point  in  time  on  the  status  of  his  program's  T&E  effort.  He  is  well 
aware  of  aill  the  technical  details,  what  might  slip,  what  has  slipped,  what 
the  time  eind  cost  impacts  are,  what  actions  axe  necessary  to  correct  situa- 
tions, etc.  There  would  appear  to  be  no  need  for  personnel  external  to  his 
organization  to  require  any  detailed  data  of  that  sort,  particularly  lest 
he  consider  himself  vulnerable  to  being  managed  by  another  layer  of  organ- 
izational codes . 

The  entity  called  the  Systems  Command  is  a corporation  within 
itself.  It  must  speak  with  a cohesive  corporate  voice  to  the  external 

world,  that  be  in  the  form  of  CHNAVMAT,  CNO,  SECNAV,  SEGDEF  or  auiy 

number  of  other  commatnds  contacted  formally  or  informsl.ly  during  the  course 
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of  dally  business.  Not  only  is  the  Systems  Commander  the  normal  chartering 
agency  for  all  of  the  Project  Offices  within  his  command,  he  is  also  the 
Heewi  of  Procuring  Activity  (HPA)  in  the  contract  arena,  and  also  owns  the 
Comptroller  as  well.  His  involvement  is  cleatr  and  he  requires  an  across  the 
board  assessment  of  the  integrated  status  of  all  his  program  responsibili- 
ties so  that  his  "corporate  voice"  is  consistent,  viable  and  credible. 

It  is  particularly  important,  therefore,  that  certain  test  and 
evaluation  information  be  mauie  available  to  the  Systems  Commander  to  support 
his  manat eTi'jnt  posture.  Particulatrly  since  seme  test  and  evaluation  mile- 
stones havG  recently  become  significant  hurdles  in  the  acquisition  process 
which  commauid  SECNAV  and  SECDEF  attention  and  need  to  be  mauie  visible  at 
the  Flag  manaigatient  level.  For  instauice.  Figure  3*1  illustrates  some  of 
these  decision  gates. 


Figure  3»1  Mandatory  T&E  Milestones 

The  ultimate  aim  at  the  end  of  Full-Scale  Engineering  Develop- 
ment is  a successful  DSARC  III  decision  and  the  releaise  of  production 
dollairs.  Before  production  funds  can  be  releaised,  current  directives  re- 
quire that  the  Project  Mauiaiger  provide  proof  to  the  Comptroller  that  the 
equipment  or  system  involved  hais  been  approved  for  service  use.  ApprovaJ. 
for  service  use  cauuiot  be  obtained  until  a successful  OPEVAL  hais  been  com- 
pleted auid  the  independent  OTieE  Agency  (COMOPTEVFOR)  has  recemmended  this 
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action.  An  OPEVAL  cannot  be  performed  until  the  equipment  or  system  Is 
"certified  ready  for  OPEVAL"  In  accordance  with  CNO  direction.  This  certi- 
fication cannot  be  requested  or  granted  until  a successful  TECHEVAL  Is 
achieved,  and  a successful  TECHEVAL  Is  dependent  upon  prior  test  eind  eval- 
loatlon  events.  So  Individual  T&E  milestones,  both  minor  auid  major,  axe 
significantly  Important  to  the  project  and  to  the  systems  ccaunand  as  a 
whole.  Failure  to  achieve  one  of  them  In  Figure  3.1  GAN  HALT  THE  ENTIRE 
ACQUISITION  PROCESS  for  a project*. 

For  this  reason  alone,  the  Systems  Commander  must  be  made 
awaxe  of  a cextaln  level  of  T&E  milestone  status.  Each  Project  Manaiger 
can  Individually  advise  the  Commander  of  the  status  of  his  particular 
program.  Significant  problems  aare  normally  highlighted  during  weekly  review 
meetings,  program  reviews,  document/correspondence  coordination  and  so 
forth.  When  a crisis  evolves,  the  problem  Is  probed  In  depth  and  potential 
solutions  receive  Intensive  management.  This,  however.  Is  not  the  level 
of  status  reporting  being  referred  to.  The  Systems  Commander  needs  a con- 
cise overview  of  totad.  status,  provided  regulaxly.  In  an  understandable 
format,  that  comm\nl cates  Intelligible  data  briefly  with  some  meaning,  and 
represents  the  total  spectrum  of  T&E  status  across  ad.1  PM  boundxles. 

The  SYSCOM  T&E  Coordinator  fxnctlon  was  established.  In  peort, 
to  provide  such  Information.  The  following  sections  discuss  his  capability 
to  perform  this  service. 

3.4.2  Scope  of  Data 

The  Chief  of  Naval.  Operations  (CNO)  Issues,  on  a quaurterly 
benls,  a document  entitled  "Index  of  CNO  Assigned  T&E  Projects",  which 
lists  all  programs  requiring  the  submission  of  a TEMP  (l.e.  AGAT  I-IIl)  (lO). 
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The  listing  provides  the  Test  and  Evaluation  Identification  Number  (TEIN) 
eissigned,  the  Program  Title • Program  Element  Number,  Project  Number  and 
OPNAV  Sponser  Code.  The  list  currently  contains  approximately  600  programs. 
In  addition,  there  are  approximately  60  ACAT  IV  programs  in  existence  in 
NAVMAT  which  require  the  preparation  of  a 'TEP.  Section  5*2  of  this  paper 
discusses  data  (milestone  events)  contained  in  Parts  I and  III  of  TEMPs 
and  TEPs  and  identifies  over  180  key  test  and  evaluation  and  acquisition 
process  data  points  which  must  be  scheduled,  coordinated,  monitored  and 
executed  for  each  program.  Simple  mathematics  indicates  that  there  are 
over  126,000  different  milestone  dates  involved  with  the  T&E  effort  within 
NAVMAT.  This  number  does  not  include  other  acquisition  milestones  not 
directly  associated  with  T&E,  nor  does  it  Include  any  lesser  importauit 
T&E  milestones,  of  which  there  are  numerous  in-house  eind  contractor  asso- 
ciated events  at  aJ.1  levels  of  design  and  development.  For  any  one  SYSCOM, 
the  number  of  KEY  T&E  milestones  could  be  50,000  to  100,000.  Quite  a number 
for  one  T&E  Coordinator  to  be  aware  of  1 1 

3.^.3  Current  T&E  Coordinator  Gambility 

If  Senior  Management  asked  the  T&E  Coordinator  to  provide  a 
particular  piece  of  detailed  engineering  information  such  asi  What  module 
failed  in  the  AN/XXX  widget  that  caused  a certain  test  to  fail,  they  axe 
probably  asking  the  wrong  man,  and  the  question  should  be  directed  toward 
the  Project  Office  vice  the  T&E  Coordinator.  Similarly,  questions  concerning 
a peurticular  milestone  Impact,  funding  implications  and  so  forth  are  more 
a PM's  responsibility  to  answer  since  the  T&E  Coordinator  cannot  be  aware 
of  all  the  details  of  everyone's  programs.  He  can,  however,  on  occasion 


act  as  a liaison  and  obtain  answers,  but  this  is  an  inefficient  way  of 


doing  business. 

If  Senior  Managanent  asks  the  T&E  Coordinator  for  status  Infor- 
mation which  cuts  across  PM  or  functional  boundrles  such  asi  How  many  TEMPs 
have  been  written,  and  how  many  more  are  required;  or  who's  having  OPEVALs 
during  the  next  quarter  - THIS  la  the  Information  that  should  be  provided 
l?y  the  T&E  Coordinator.  He  has  all  the  data  with  which  to  answer  these 
questions.  They  axe  contained  In  the  TEMPs  aind  TEPs  In  the  safe  In  the 
comer  because  the  directives  have  stated  that  he  shall  be  the  "repository" 
for  them  all.  Unfortunately,  In  order  to  be  able  to  siscertaln  how  many 
OPEVALs  are  scheduled,  he  will  have  to  manually  search  each  TEMP,  sceui 
60,000  dates  for  the  100  asked  for,  list  the  Information,  consolidate  It, 
write  It  up  In  matrix  euid  memo  form  and  forward  It  to  the  questioner. 

This  effort.  In  ccsnblnatlon  with  phonecalls  to  knowledgable  PM  personnel 
and  other  sources  of  command  Information,  could  conceivably  taike  care  of 
his  entire  morning.  The  jiftemoon  could  also  disappear  If  another  question 
concerning  the  number  of  DT-IIa's  we've  had  this  year  Is  eisked. 

The  point  Is  that  a manuail  search  of  data  each  time  a question 
Is  asked  takes  time.  There  Is  no  ccmunon  data  base  that  Is  continually  up- 
dated on  an  as-occurlng  basis,  and  no  automatic  means  of  generating  sched- 
uled reports  which  might  provide  the  necessary  Information  without  being 
aisked.  The  T&E  Coordinator,  like  everyone  else,  manages  his  job  by  excep- 
tion and  Is  not  afforded  the  luxury  of  executing  planned  functions  at 
specific  times  - all  on  schedule.  His  dally  workload  Is  based  on  a specific 
nvunber  of  documents  passing  through  each  week,  a few  unanticipated  "vihat-lfs, 
and  a normal  baseline  of  dally  tasks.  Rarely  Is  he  required  to  perform 
all  his  assigned  responsibilities  In  parallel;  nor  do  all  the  TEMPs  come 
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in  at  once  for  review,  nor  does  he  receive  eui  inordinate  amount  of  ques- 
tions each  day. 

The  purpose  of  this  discussion  is  to  point  out  that  the  T&E 
Coordinator  has  a daily  job  to  do.  He  performs  his  functions  as  outlined 
in  Appendix  B on  am  as-required  basis  and  manages  his  daily  workload  in 
accordance  with  the  existing  priorities  at  the  time.  What  he  is  NOT  doing, 
however,  is  manuadly  generating  basic  status  reports  up  the  chain  of 
command  frcm  the  data  contained  in  the  TEMPS  filed  in  his  safe.  His  status 
reporting  capability  is  reactionary  - responding  to  specific  questions  - 
probably  asked  because  of  the  absence  of  the  status  report  in  the  first 
place.  The  T&E  Coordinator  (normally  one  person)  has  not  the  time  or 
capability  to  collect  and  manually  structure  a meaningful  group  of  T&E 
status  reports  on  any  regular  basis. 

3.^.^  The  Proposed  Solution 

There  is  a solution.  As  Indicated  above,  all  the  required  data 
is  available  in  one  spot.  By  Initially  taking  the  time  (with  external 
help)  to  rer'ormat  the  information  in  the  existing  documents  into  a computer 
data  base,  and  then  implementing  procedures  for  Insuring  that  the  develop- 
ment of  additional  database  information  is  accomplished  each  time  a new 
TEMP  or  TEP  is  prepared  for  review  and  submission.  Step  One  of  the  solution 
has  been  completed.  Step  Two  invol'^es  a computer  program  which  essentially 
"sorts  and  prints"  data  onto  prmcribed  output  report  formats  which  can  be 
further  distributed  to  cognizant  codes. 

Before  this  paper  proceeds  into  the  details  of  a proposed  infor- 
mation system,  it  is  recognized  that  some  reaiders  have  a natural  aversion 
to  any  form  of  computerized  "MIS"  due  to  jjrior  be«i  experiences  or  a number 
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of  other  reasons,  'nierefore,  Section  4.0  will  explain  the  author's  percep- 
tion of  an  "Infomation  System"  as  opposed  to  a normally  envisioned  Manage- 
ment Information  System,  and  will  discuss,  in  more  detail,  the  methodology 
behind  the  proposed  TAB  computer  system.  Section  5.0  will  then  define  spe- 
cific data  Inputs  and  output  formats  achievable  with  such  a system. 
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4.0  DEVELOPMENT  OF  AN  INFORMAnON  SYSTEM 

4.1  OVERVIEW  CF  MANAGEMENT  INFORMATION  SYSTEMS  (MIS) 

To  a number  of  people,  the  term  "MIS"  is  an  unpopuleu:  word.  It  is  syn- 
oncmous  with  a stack  of  smelly  paper  three  feet  high  that  comes  into  the 
office  once  a week  and  piles  up  in  the  comer  at  the  rate  of  twelve  feet 
per  month.  Ihe  information  is  not  in  a form  which  can  be  iTeadily  used,  and 
to  manually  translate  it  would  take  three  times  as  many  people  twice  as 
long!  so  therefore  it's  useless.  "MIS"  represents  a computer  down  the  hall 
that  costs  $100,000,  which  could  have  paid  for  several  extra  people  in  the 
office  because  you  now  need  them  to  help  feed  the  extra  input  data  to  the 
machine  each  day. 

Unfortunately,  in  this  type  of  situation,  the  observer  is  correct. 

If  the  computer  is  not  providing  the  right  information,  or  is  providing  the 
right  information  in  the  wrong  format,  it  won't  be  used.  Unless  a "MIS"  is 
tailored  to  a specific  person,  teisk  or  job,  suid  the  user  has  had  a major 
voice  in  its  development,  the  outputs  will  continue  to  be  office  door  stops 
and  coffee  cup  coasters  for  the  life  of  the  system. 

Once  tailored  to  the  job,  a "MIS"  can  be  extremely  helpful  in  alle- 
viating a number  of  cmnbersome,  repetitive,  menial  tasks  in  the  office;  elim- 
inating some  of  the  time-eating  jobs  and  releasing  the  workers  to  perform 
more  important  functional  tasks^.^,^  a sense,  the  "MIS"  mechanizes  a certain 
portion  of  the  office  task.  It  processes  information  much  faster  than  human 
mEUiipulatlon  (provided  that  the  bureaucratic  administrative  handling  doesn't 
slow  down  the  input/output  process),  data  is  esisy  to  update  if  the  proper 
forms  and  procedures  are  Implemented,  and  a niomber  of  "what  if"  games  can 
be  played  with  ease  because  various  permutations  of  the  data  base  can  be 
extracted  and  analyzed  at  the  push  of  a button,  provided  the  computer  hais 
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been  properly  programmed  in  advance. 

On  the  other  side  of  the  coin,  however,  are  factors  such  as  cost  (both 
of  hardware  and  software),  hardware  maintenance,  continuing  requirements 
for  program  changes  (back  to  cost  againl),  logistics  considerations  such  as 
ccMnputer  location,  time  sharing,  acquisition  of  supplies  and  so  forth.  In 
Euidition,  there  is  a tremendous  communication  barrier ^ That  is,  most  people 
just  can't  walk  up  to  a ccanputer  2uid  staurt  talking  to  it.  A knowledgeable 
Interface  must  be  sought  in  the  form  of  an  operator  or  programmer,  and  the 
user  must  transmit  his  desires  properly,  be  understood  by  this  interface, 
and  understand  what  he  gets  back  in  return  (often  in  a "foreign"  language) . 
Otherwise,  the  whole  exercise  is  doomed  to  failure.  Again,  the  major  disad- 
vantage to  a ccmputerized  "MIS"  is  the  inherent  human  resistance  to  letting 
a machine  do  his  thinking  and  do  part  of  his  job,  lest  he  lose  it.  There  is 
a general  mistrust  of  any  syston  that  is  not  under  the  complete  control  of 
the  immediate  supervisor  or  mansiger. 

In  euidition,  most  people  are  aware  of  the  work  involved  on  the  input 
side  of  the  computer  like  collecting  data,  structuring  it,  translating  it 
into  proper  computer  terminology,  transcribing  it  and  transmitting  it  to 
the  machine.  In  fact,  the  initial  data  base  effort  is  often  more  difficult 
and  time  consuming  than  doing  the  entire  function  manually  (C^GEI).  What 
people  fail  to  realize  is  that,  once  over  that  initial  hump,  the  flexibil- 
ity, diversity,  speed  and  cost  savings  on  the  output  side  far  outweigh  any 
further  maintenance  data  efforts  established.  Once  the  system  becomes 
accepted  and  routine,  the  ability  to  generate  any  number  of  structured  re- 
ports at  the  push  of  a button  dilutes  relative  concerns  about  input  work. 

Most  managers  think  of  the  initial  input  work  when  approached  with  a new 
"MIS"  and  do  not  appreciate  the  output  benefits  once  the  initial  work  is  done. 
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4.2  INFORMATION  SYSTEM  VERSUS  MANAGEMENT  INFORMATION  SYSTEM 


The  reader  has  perhaps  noticed  that  any  reference  to  "MIS"  has  been 
written  in  quotation  matrks.  In  the  author's  opinion  the  word  "management" 
in  "MIS"  is  a misnomer  (no  pun  intendedi).  MANAGEMENT  INFORMATION  SYSTEMS 
DON’T  MANAGE  - PEOPLE  DOl  The  "MIS"  only  provides  data  in  a form  which 
meikes  it  easier  for  the  manager  to  manage;  not  to  automatically  do  his  job 
for  him.  For  this  reason  the  term  "MIS"  has  done  a great  disservice  to  its 
concept  and  has  permanently  alienated  a number  of  potential  users  who  were 
disappointed  because  their  former  systems  didn’t  "manage"  like  they  were 
supposed  to. 

This  paper  will  emphasize  the  "IS"  eispect  - an  Information  ^stem  for 
test  and  ev^LLuation;  not  a Mauiagement  Information  System.  The  ratloneLle 
for  this  statement  is  the  foundation  for  the  purpose  of  this  paper  - to 
provide  a consolidated  listing  of  TisE  milestone  data  in  easily  understand- 
able form.  A large  number  of  our  management  problems  (aside  frcm  manpower, 
money  ^uld  time)  are  a result  of  incorrect  or  untimely  information.  How  msiny 
times  has  a supervisor  asked  a comprehensive  question  and  it  has  taken 
several  hours  (or  days)  to  answer.  Vfhy?  Because  Mary  Jones  had  part  of  the 
answer  in  her  head;  Tom  Smith  entered  part  of  the  answer  in  Book  "A"  leist 
week  but  got  diverted  before  he  could  update  Book  "B";  and  Billy  Jean  used 
Book  "B"  to  answer  the  question.  The  answer  was  therefore  out  of  date  and 
incomplete,  but  the  data  was  in  the  office  IN  ONE  FORM  OR  ANOTHER. 

This  situation  exists  because  information  filters  into  an  office  via 
many  paths,  and  it  is  received  and  recorded  'ky  as  many  means.  Information 
is  documented  in  primary  places  where  it  belong  (logs,  records,  books, 
files,  etc),  seldom  in  secondary  (backup)  files,  and  never  in  other  places. 
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However,  all  these  places  come  into  play  at  one  time  or  another  in  develop-  K'I;' 

ing  information  to  leave  the  office.  There  is  never  a complete  satisfaction 
or  high  level  of  confidence  that  the  information  is  totally  correct  at  any 
time  because  the  office  personnel  know  that  this  data  diffusion  problem 
exists.  They  strive  to  minimize  the  error  of  the  output,  hoping  that  some 
important  impacting  data  has  not  been  overlooked  in  "Book  X".  Office  person- 
nel wish  that  all  their  information  could  be  collected  one  person  in  one 
big  pot  in  the  center  of  the  office,  sind  everytime  a task  was  performed,  the 
up-to-date  correct  data  would  always  be  extracted  frcm  the  pot  with  near 
100^  confidence. 

The  wish  is  for  an  information  system  - a place  where  all  correct  data 
is  stored  and  extracted.  The  pot  is  a computer.  The  beauty  of  the  system  is 
that  a piece  of  data  (assuming  it  is  a correct  piece  of  data)  is  entered 
once  into  the  memory  and  no  matter  who,  for  what  purpose,  for  how  many 
times,  extracts  a number  of  pieces  of  data  for  some  reaison,  the  information 
received  is  always  a correct  reflection  of  the  data  in  the  memory.  This 
advantage  may  not  appear  too  exciting  until  one  has  experienced  this  type 
of  exercise  manually  a number  of  times  with  continually  varying  levels  of 
input  data  credibility. 
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4.3  TEST  AND  EVALUATION  INFORMATION  SYSTEM  REQUIREMENT 


Section  3*4  discussed  the  need  to  coordinate  and  mauiipulate  a number 
of  data  elements  contained  in  the  collection  of  Test  and  Evaluation  Master 
Plans  (TEMPs) . It  described  a situation  wherein  a T&E  Coordinator  position 
has  been  established  in  a staff  capacity  to  the  System  Commander  with  a 
responsibility  to  provide  detailed  summary,  coordinative  and  advisory  infor- 
mation on  elements  of  individual  Project  Manager  efforts  for  which  the  T&E 
Coordinator  has  minimum  day  to  day  responsibility  or  authority  over.  Section 
3.4  additionally  pointed  out  that  available  T&E  Coordinator  manpower  versus 
perceived  Coordinator  functions  could  be  in  serious  inbalance  depending  on 
the  circumstances  and  frequency  of  desired  information  by  senior  manage- 
ment. 

The  solution  to  this  situation  was  just  discussed,  and  it  is  basically 

a simple  onei  Create  the  "pot",  collect  the  inputs,  and  format  and  dissem- 
inate the  outputs.  It  is  not  simple  from  the  standpoint  of  the  development 

of  the  system  itself,  but  from  the  fact  that  a majority  of  the  data  inputs 

are  already  available  to  the  coordinator  in  a semi-formatted  form  (TEMPs 
and  TEPs),  and  the  responsibility  for  maintaining  updated  T&E  data  is  alrea- 
dy being  performed  by  the  Project  Manager.  What  is  needed  is  a scheme  to 
tie  this  all  together  - the  need,  the  means  and  the  might'.  A partial  plan 
of  action  could  be  as  follows 1 

• Define  the  requirement  - which  th5s  paper 
serves  to  accomplish 

• Structure  the  beisic  input  emd  output  data 
needs  - again,  this  paper  refers 

• Obtain  approval  within  the  canmand  for  the 
implementation  of  such  an  information 
system 

• Obtain  orgsuiizational  acceptance  of  the 
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concept.  The  system  won't  survive  if  the 
players  don't  play  aind  believe  in  the 
potential  benefits 

• Decide  on  what  type  of  computer  support 
to  use,  such  as 

- Leased  (computer  external  to  command) 

- Time  Share  (with  existing  in-house 
system) 

- Buy  (dedicated  hardware  for  stamd- 
alone  system) 

• Obtain  funds  to  support  the  system 

• Identify  programming  capability  and 
conmence  communications 

• Write  implementing  instructions  to  identify 
necessairy  responsibilities  and  procedures 
once  the  system  is  operational 

• Prepare  input  data  worksheets  for  all  TEMP 
data,  collect  missing  data  from  other 
sources  and  insure  that  follow-on  data  is 
to  be  autcmatically  included  in  future 
inputs 

• Insure  that  the  test  and  evaluation  personnel 
in  the  functional  and  project  managanent 
organizations  are  involved  in  eill  evolutions. 
It  is  essentiaJ-ly  their  syston. 

• Implement  auid  maintain  the  system 

• Obtain  feedback  on  the  system  and  report 
effectiveness  and  make  necessary  changes 
accordingly 
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^.4  SUMMARY 

Computers  axe  fast  becoming  a way  of  life.  Within  the  next  ten  to  fif- 
teen yeatrs  they  will  be  as  common  as  a calculator  or  electric  typewriter, 
both  in  the  office  aJid  the  home.  Already  the  consumer  msirket  is  beginning 
to  show  the  signs  and  is  providing  desk  top  commercial  ccanputers  for  less 
than  $1000.  Even  Heathkit  and  Radio  Shack  have  entered  the  market  this  year. 
Government  and  the  business  world  axe  being  ilooded  with  numerous  systems, 
formerly  C20.1ed  "Programmable  Calculators"  which  axe  giving  individual  off- 
ices their  own  in-house  computer  capability  right  on  the  mamsiger's  desk. 

They  are  programmed  in  Basic  Language  and  often  in  a form  which  leads  the 
layman  by  the  hand  through  a step  ly  step  execution  of  the  program.  Although 
rigid  guidelines  have  been  imposed  within  OSD  concerning  the  need,  jxasti- 
fication  and  use  of  computer  systems,  the  day  will  come  when  manager's  will 
be  able  to  afford,  and  choose  to  use,  personal  systems  to  help  him  do  his 
job  better,  as  he  does  now  by  carrying  around  a $250  Hewlett-Packard  calc- 
ulator in  his  briefcase. 

The  choice  of  what  type  of  computer  system  to  use  in  implementing  a 
test  and  evaJ.uation  information  system  must  be  made  'ty  the  individual  ccmi- 
mand  based  upon  current  policy,  attitudes,  manpower,  money  available  and  the 
extent  and  methods  currently  employed  in  utilizing  existing  computer  systems. 
Whatever  system  is  chosen,  the  concepts  in  this  paper  should  be  further 
modified  and  tailored  to  the  individvial  characteristics  of  the  machine. 

In  Section  5*0  a particiilar  test  and  evaluation  information  system 
is  proposed.  Data  inputs,  as  extracted  from  the  Test  and  Evaluation  Master 
Plans  (TEMPs)  are  defined  and  consolidated  on  aui  Input  Data  Worksheet  for 
entry  into  the  computer.  Data  output  reports  will  be  categorized  and  for- 
matted into  specific  T&E  milestone  information  packages. 
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5.0  DEVELOPMENT  OF  A TEST  AND  EVALUATION  INFORMATION  SYSTEM 

5.1  Introduction 

5.1.1  General 

In  the  preceding  section  aui  effort  was  made  to  emphasize  the 
flexibility  of  a computer  system  to  print  various  tailored  outputs  when  so 
programmed..  Such  a system  should  provide  useable  information  in  useable 
form;  that  is,  provide  data  which  will  enable  maJiagers  to  maJiage  better 
without  the  system  attempting  to  mauiage  the  meinager.  This  section  proposes 
a framework  information  system  for  collecting,  storing,  sorting,  printing 
and  distributing  test  and  evaluation  data  for  use  by  System  Command  manaige- 
ment  personnel  and  Program/Acquisition  managers. 

In  developing  sin  information  system,  there  are  a number  of  ac- 
tions that  appear  to  happen  all  at  once.  As  the  requirement  is  recognized 
euid  the  problem  is  scoped  and  tailored,  ideas  axe  developed  concerning  the 
type  of  data  elements  to  be  considered  as  inputs  to  the  system  and  their 
sources.  At  the  seune  time  "hexd  copy"  output  formats  are  being  formulated 
to  display  the  information  in  the  desired  meinner.  The  input-output  process 
is  iterative.  Some  report  formats  are  generated  because  of  the  availability 
of  input  data,  and  some  input  data  is  sought  because  of  the  requirements  of 
the  output  report.  The  process  expands,  reduces  and  tailors  the  computer 
program  requirement  to  fit  the  exact  needs  of  the  user.  It  is  not  possible 
to  commit  this  Iterative  thinking  process  to  paper.  Although  this  section 
first  develops  a concept  of  input  data  followed  by  a concept  of  output 
report  formats,  the  two  concepts  were  mutually  derived. 

5.1.2  Input  Gcxiunents 

InTJuts  to  the  information  system  axe  primaxily  extracted  from 
TEMPs  and  TEPs,  and  the  data  is  concentrated  in  two  areas  of  the  documents 
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SIS  follows  I 


PART  I - Administration  Information 

Contains  data  concerning  the 
identification  of  the  program, 
points  of  contact  and  applicable 
documentation . 

PART  III  - Integrated  Schedule 

Consists  of  one  page  (normally  a 
fold-out)  displaying  the  integrated 
time-sequencing  of  all  test  and 
evaluation  and  related  key  events 
in  the  acquisition  decision  making 
process . 

Three  administration  categories  and  twelve  integrated  schedule  categories 
aire  further  broken  down  and  described  in  Section  5*2  (input  Data).  To  fa- 
cilitate the  treuisfer  of  these  data  points  from  the  TEMP/TEP  documentation 
into  the  computer,  Section  5*2  further  describes  an  "Input  Data  Worksheet", 
a sample  of  which  is  provided  as  Appendix  C to  this  report. 

The  manipulation  of  input  data  must  be  flexible.  No  programs 
are  Identical  and  there  is  no  set  of  totally  common  milestones  aunong  them. 
There  aire,  however,  a number  of  core  milestones  such  as  DSARCs,  OPEVALS, 
TECHEVALS  auid  the  like  which  are  common  to  most  programs.  Although  the 
definitions  of  input  data  include  a varying  number  of  different  types  of 
data,  these  core  milestones  will  be  used  most  frequently  in  output  reports. 

In  reviewing  a number  of  TEMPs,  it  is  evident  that  the  level  of 
detail  of  recorded  milestones  and  completeness  of  entries  vauries  from  doc- 
ument to  document  depending  upon  the  status  of  the  program  at  the  time  the 
TEMP  was  written.  Secondary  sources  of  Information  must  be  used  in  these 
cases  to  fill  in  the  missing  information. 

Lastly,  this  paper  does  not  attempt  to  enter  the  realm  of  the 
programmer  and  the  programming  language  required  to  make  the  system  work. 


• 

(Part  II 
not  • 
applicable) 

• 
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Data  must  be  numerically  keyed  and  properly  arranged  within  a matrlxed  mem- 
ory bsuik  so  that  it  can  be  easily  found,  recalled,  inter-related  with  other 
data  and  printed  in  preprogrammed  formats.  Section  5*2  does  not  address  these 
programming  requirements  when  describing  input  data. 

5.1.3  Output  Comments 

Once  the  input  parameters  have  been  defined.  Section  5*3  defines 
the  veltIous  types  of  reports  that  could  be  generated  as  outputs.  These  reports 
basically  fall  into  four  categories! 

• Box  Scores 

• Program  Milestone  Listings 

• Major  Milestone  Listings 

• Document  Update  Status 

In  auidltion  to  these  types  of  reports,  the  information  within 
each  category  csui  be  sorted  and  printed  in  a number  of  different  groupings 
depending  on  the  form  in  which  the  information  is  desired  by  senior  manage- 
ment. For  Instauice,  data  cam  be  grouped  and  printed  by« 

• Chronological  Order  of  Milestone  Data 

• Type  of  Acquisition  Category  (A CAT  I,  II, 

III,  or  IV) 

• Program/Acquisition  Management  Organization 

• Program  Elements 

• T&E  Identification  Numbers  (TEINs) 

Any  combination  of  grbliplngs  is  possible,  amd  the  peurticulax  order  of  data 
presentation  within  the  primatry  groupings  cam  also  be  vauried.  For  instance. 
Figure  5*1  illustrating  a patrtial  report  on  TEMPs  lists  the  documents  in 
accordamce  with  their  ACAT,  and  further  orders  the  list  by  PM  orgamization 
and  ascending  order  of  TEIN. 
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AGAT  I I 


AGAT  I 1 PMX  103 

TEIN  23 

TEIN  45 

PMX  105 

TEIN  21 

TEIN  142 

AGAT  III  PMX  136 

TEIN  102 

TEIN  568 

TEIN  874 

Figure 

5.1  TEMP  Report 

The  seime  type  of  Information  could  very  well  have  been  presented  in  a com- 
pletely different  format  as  shown  by  Figure  5*2  below. 


PM  ORG 

PROG  ELM 

AGAT 

TEIN 

PMX  103 

45876N 

I 

45 

79352N 

I 

23 

PMX  105 

24269N 

I 

21 

6344IN 

I 

142 

PMX  136 

47446N 

II 

568 

61332N 

II 

102 

73^5N 

II 

874 

Figure  5-2  TEMP  Report 

Here,  the  same  data  was  provided  in  columns,  but  ordered  by  acsending  se- 
^ence  of  project  organization.  Within  that  grouping,  data  was  listed  hy 
ascending  order  of  program  element  number.  There  is  a large  degree  of  flex- 
ibility in  the  types  of  output  forms  available  using  the  same  input  data  in 
memory,  suid  reports  can  be  structured  in  any  manner  depending  opun  what  maui- 
agement  wsuits  to  primarily  focus  on. 

The  availability  of  so  many  format  permutations  drives  the  system 
towaird  a set  of  "standard  displays"  which  czui  then  be  assigned  coded  number 
identifications  to  facilitate  their  identification  and  cataloguing.  In  ad- 
dition, these  output  formats  <»n  be  programmed  in  advauice  and  cauLled  out  of 
the  computer  as  often  or  as  little  as  required.  The  entire  system  and  pro- 
cedure should  be  designed  to  minimize  operator  and  management  personnel  time. 
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Another  feature  of  the  output  system  is  that  the  tune  required 
to  prepare  and  distribute  reports  is  also  minimized  since  the  computer  can 
be  programmed  to  print  the  cover  memornuidum,  complete  with  datei  from,  to, 
subject  and  distribution  codes.  The  T&E  Coordinator  need  only  reproduce  the 
hard  copy  output  of  the  computer  and  route  to  the  distribution  codes  involv- 
ed. 

Sample  output  reports  for  each  type  of  proposed  format  have  been 
developed  and  included  in  Appendices  D through  K,  and  are  discussed  in  more 
detail  in  Section  5*3* 

5.1.4  Introduction  Summary 

This  introduction  has  attempted  to  give  the  reeider  a brief  over- 
view of  the  Information  process  and  explain  the  purpose  of  the  detailed 
listing  of  data  elements  in  the  following  sections.  The  author  firmly  be- 
lieves that,  once  the  initial  effort  of  computer  programming  and  data  collec- 
tion axe  completed  and  the  system  begins  normal  maintenance  and  update  oper- 
ations, the  ewivamtages  gained  in  having  aui  instant  recall  of  a number  of 
T&E  status  reports  which  reflect  real-time  updated  input  data,  can  be  a 
tremendous  mainagement  aid  to  the  Systems  Commands  in  keeping  tabs  on  the  vo- 
Ivanous  numbers  of  T&E  events  occuring  in  Project  Management  and  Acquisition 
organizations. 
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5.2  INHJT  DATA 

5.2.1  Administrative  Data 

As  defined  in  the  previous  section,  administrative  data  is  gen- 
erally found  in  Part  I of  the  TEMP  or  TEP.  The  administrative  data  is  the 
key  to  Identifying  the  program  itself  and  the  TEMP  in  which  the  milestone 
data  is  documented.  Most  output  formats  will  tend  to  rely  on  four  or  five 
of  these  administrative  parameters  to  group  data.  The  input  data  below  will 
be  required  for  this  T&E  information  system. 

5. 2. 1.1  Identification  Data 

• TEIN 

A "Test  and  Evaluation  Identification  Number", 
normally  assigned  by  CNO  to  each  acquisition 
program  for  the  life  of  the  program.  It  also 
becomes  the  TEMP  number  which  is  usually  three 
digits  such  as  TEIN-383. 

• Full  Program  Title 

The  title  appearing  in  the  TEMP.  There  are  many 
forms  of  titles  (Program  Element  Title,  Program 
Short  Title,  Abbreviated  Titles,  etc).  To  be 
consistent,  the  data  should  match  the  TEMP 
title . 

• Program  Element  Number 

An  alpha-numeric  number  such  as  "6231 IN". 

• Pro.iect  Nianber 

A number  of  vaurying  length  and  description  due 
to  a recent  change  in  numbering  systems.  Some 
TEMPs  cite  both  old  auid  new  numbers.  For  con- 
sistency, the  new  nomenclature  should  be  used. 

Where  no  number  has  been  assigned,  or  is  un- 
known, a dummy  number  should  be  inserted  in  the 
input. 

• Acquisition  Category 
AGAT  I,  II,  III  or  IV 

• RDT&E  Fund  Level 

Mauclmum  rounded-off  dollau*  amount  of  the  RDT&E 
portion  of  the  program. 


• Procurement  Fund  Level 


Maximum  rounded-off  dollar  amount  of  the  procurement 
portion  of  the  program.  (Other  funding  such  as  O&MN, 
MILGON  etc , , not  addressed  in  TEMP  Part  l) . 

5. 2. 1.2  Points  of  Contact 

The  Name,  Orgauiizational  Code  ajid  Telephone  Number  for 
each  of  the  following  prime  points  of  contact  should  be  included i 

• GNO  Program  Sponsor 

• Project  Mamager 

• Acquisition  Mauiager 

• Test  Director 

5 • 2 . 1 . 3 Documentation  Data 

The  Identifying  Number  and  Date  of  the  last  version  of 
the  following  documents  should  be  Included i 

• Program  Memorandum  (PM) 

• Navy  Decision  Coordinating  Paper  (NDCP) 

• Decision  Coordinating  Paper  (DCP) 

• Operational  Requirement  (OR) 

• Development  Proposal  (DP) 

• Science  smd  Technology  Objectives  (S&TO) 

• Test  and  Evaluation  Master  Plan  (TEMP)  or 
Test  2Uid  Evaluation  Plan  (TEP)  - Both 
original  date  and  latest  revision. 

This  canpletes  the  scope  of  administrative  type  input  data.  These  Inputs 
are  consolidated  on  an  "Input  Data  Worksheet"  discussed  in  Section  5»2.3 
and  included  as  Appendix  C to  this  report. 

5.2.2  Integrated  Schedule  Data 

Specific  test  and  evaluation  milestone  data  is  found  in  P^trt  III, 
"Integrated  Schedule"  portion  of  the  TEMP  or  TEP.  The  milestone  data  is  group- 
ed into  twelve  categories  2Uid,  within  each  category,  the  milestone  dates  are 
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sjtread  throughout  a 5 to  8 fiscal  year  range.  For  the  purposes  of  this  infor- 
mation system,  input  data  will  be  collected  in  accordance  with  the  grouping 
of  these  categories  vice  attempting  to  group  them  by  chronological  order  or 
by  acquisition  phases.  The  relative  phasing  (integration)  of  all  milestones 
will  be  accomplished  by  the  computer  program  itself.  It  is  Important  that 
the  input  data  collection  process  be  convenient  and  easily  tiunslatable  from 
the  TEMP  or  TEP. 

Milestone  events  will  either  be  characterized  as  "point"  events, 
with  only  one  date  associated  with  the  effort  (such  as  DSARC)  or  will  have 
a start  and  complete  date  associated  with  an  effort  of  longer  scope  (such 
as  OPEVAL) . The  best  information  should  be  used  In  any  case.  The  following 
sections  describe  the  types  of  milestone  data  that  will  be  required  for  the 
information  system. 

5.2.2. 1 Major  Milestones 

• Milestone  ^ - Program  Initiation 

• (N)SARCs  - I,  II,  III 

• DSARCS  - I,  II,  III 

• Technical.  Reviews 

- System  Requirement  Review  (SRR) 

- System  Design  Review  (SDR) 

- Preliminary  Design  Review  (PDR) 

- GriticaJ.  Design  Review  (CDR) 

- Functional.  Configuration  Audit  (FGA) 

- Physical  Configuration  Audit  (PGA) 

- Production  Reauiiness  Review  (PRR) 

• Major  Equipment  Installations 

- Prototype 

- Production 

• Certification  of  Reaidiness  for  OPEVAL 

- Requested 

- Granted 


• Provisional  Approval  for  Service  Use  (PASU) 

- Requested  and  Granted  Dates 

• Approval  for  Service  Use  (ASU) 

- Requested  and  Granted  Dates 

• Prosrao  Planning 

- Procurement  Plan  (PP) 

- Program  Management  Plan  (PMP) 

- System  Engineering  Management  Plan  (SEMP) 

- Integrated  Logistics  Support  Plaui  (ILSP) 

- Source  Selection  Plan  (SSP) 

- Operational  Requirement  (OR) 

- Development  Specification 

- System  Specification 

- Product  Specification 

- Material  and  Process  Specifications 

- Work  Breakdown  Structure  (WBS) 

- Procurement  Request  (PR) 

- Request  for  Proposal  (RFP) 

• Long-Leeid  Procurement  Items  (including  Capital 
Investment  Items) 

• Initial  Operational  Capability  (lOC) 

• Full  Operational  Capability  (FOC) 

• Other  Major  Milestones  as  applicable  to 
individual  programs. 

5 • 2 . 2 . 2 Contract  Dates 

• Applicable  Procuronent  and  RFP  Planning  above 

• Determination  auid  Finding  (d/f) 

• Response 

• Negotiation 

• Contract  Awards 

- Conceptual 

- Demonstration  and  Validation 

- Full  Scale  Engineering  Development 

- Pilot  Production  Model 

- Production 

• Applicable  Source  Selections 
5. 2. 2. 3 DCP  or  NDCP 

• Scheduled  Updates  (Annually) 

• Updates  Required  Prior  to  NSARC/DSARC 
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1 


5. 2. 2.4  TEMP 


• Scheduled  Updates  (Annually) 

• Updates  Required  Prior  to  NSARC/DSARC 

5. 2. 2. 5  Test  Articles 

• Qualified  Farts  List  (QPL) 

• Brassboard 

• ADM  Delivery 

• ADM  Test 

• EDM  Delivery 

• EDM  Test 


5.2.2. 6  DT&E 

• DT-I 

- Lab 

- Shore 

- At  Sea 

• DT-IIa 

• DT-IIb 

• DT-IIIa 

• DT-IIIb  (TECaiEVAL) 

• Other  specific  tests  as  applicable, 

such  as  unique  contractor  demonstrations, 
acceptance  tests,  joint  testing,  etc) 

5.2.2.?  OT&E 

• OT-I 

• OT-IIa 

• OT-IIb 

• OT-IIIa 

• OT-IIIb  (OPEVAL) 

• OT-IV  (as  required) 

• OT-V  (as  required) 

5. 2. 2. 8 PATftE 


5. 2. 2. 9 DTiSsE  Test  Plan  (Staurt  - Draft  Due  - Final  Due) 

• DT-I 

• DT-II 

• DT-IIIa 

• DT-IIIb  (TECHEVAL) 

5.2.2.10  OT&E  Test  Plan  (Start  - Draft  Due  - Final  Due) 

• OT-I 

• OT-II 

• OT-IIIa 

• OT-IIIb  (OPEVAL) 

5.2.2.11  DT&E  Report  (Start  - Draft  Due  - Final  Due) 

• DT-I 

• DT-II 

• DT-IIIa 

• DT-IIIb  (TECHEVAL) 

5.2.2.12  OT&E  Report  (Staurt  - Draft  Due  - Final  Due) 

• OT-I 

• OT-II 

• OT-IIIa 

• Quick  Look  Report 

• OT-IIIb  (OPEVAL) 

This  completes  the  scope  of  Integrated  schedule  milestone  input  data.  These 
inputs  aire  aidditionally  consolidated  on  the  following  "Input  Data  Worksheet" . 
5.2.3  Input  Data  Worksheet 

The  amount  of  data  described  above  appears  ominous,  but  most 
of  it  is  contained  in  only  two  or  three  pages  of  the  TEMP/TEIP  auid  a transfer 
of  this  information  should  be  relatively  easy.  In  order  to  further  facilitate 
such  a transfer  into  the  ccanputer  data  base,  and  to  assist  in  providing  the 
required  information  initially  as  new  TEMPs/TEIPs  are  being  sulmitted,  an 
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Input  Data  Worksheet  has  been  developed  and  is  provided  as  Appendix  C to 
this  paper. 

Essentially,  the  worksheet  groups  the  required  data  into  sections 
as  was  done  in  Section  5-2.2  above,  but  in  addition  it  provides  blaink  spaces 
in  which  to  write  the  applicable  data.  This  creates  a level  of  stauidardi2a- 
tion  and  consistency  to  the  input  process.  A sample  of  the  worksheet  is 
illustrated  in  Figure  5-3  below. 

I - ADMINISTRATIVE  DATA 

1.  TEINi 

2.  Full  Program  Title t 


3.  Program  Element  Number i 

4.  Project  Number I 

Figure  5-3  Sample  Input  Data  Worksheet 

If  the  data  required  by  the  worksheet  blanks  is  not  applicable 
to  the  particular  program  being  documented,  the  term  "NA"  should  be  entered 
on  the  first  two  blanks  of  the  data  line.  If  the  data  is  temporarily  not 
available  (but  will  be  provided  at  a later  time),  the  term  "TBP"  should  be 
entered  on  the  first  three  blanks  of  the  data  line.  Dpites  should  be  entered 
with  a two  digit  day,  a three  letter  month,  auid  two  digit  yeatr  (DD  MMM  YY) 
such  as  05  NOV  78. 

When  the  input  data  has  been  entered  into  the  computer,  it  is 
stored  in  memory  and  available  to  be  sorted,  reformatted  and  printed  in 
specific  report  outputs  which  is  the  subject  of  Section  5.3  which  follows. 
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5.3 


OUTPUT  DATA 


5.3.1  Report  Categories 

This  section  will  discuss  four  types  of  report  categories  which 
would  be  of  interest  to  Systems  Command  personnel  in  managing  TEMPs,  TEPs, 
Approval  for  Service  Use  status  and  T&E  milestone  status  in  general.  The 
parenthesis  after  the  title  indicates  the  subsection  of  this  report  which 
discusses  the  format  in  more  detail,  and  the  appendix  which  provides  sample 
T&E  Coordinator  reports. 

• Box  Scores  (Section  5.3.6  - Appendices  D and  E) 

General  overviews  of  status. 

• Program  Milestone  Listings  (Section  5.3.7  ~ Appendix  F) 

Chronological  listing  of  all  milestones  within  a particular 
program.  Essentially  a phased  ordering  of  all  TEMP/TEP 
input  data. 

• Ma.ior  Milestone  Listings  (Section  5.3.8  - Appendices  G - J) 

Selected  categories  of  signiflcaiit  milestones  such  as 
DSARCS,  OPEVALS,  TECHEVALS  and  so  forth. 

• Document  Update  Status  (Section  5.3.9  - Appendix  K) 

Computes  required  update  dates  for  TEMPs,  TEPs  and  DCPs 
based  on  annual  and  major  decision  requirements.  Can  be 
used  for  amy  other  document  update  as  well. 

5.3.2  Sort  Groupings 

Within  each  report  category  above,  there  are  a number  of  ways 
the  information  can  be  sorted,  printed  and  presented  to  the  mauiager  for 
review  and  interpretation.  In  fact,  it  is  possible  to  pick  any  combination 
of  data  elements  and  prescribe  a printout  based  on  that  selected  grouping. 
Because  of  the  number  of  permutations  Involved,  a limited  number  of  data 
sort  groupings  have  been  selected  as  being  the  most  representative  of  the 
types  of  maJiually  derived  lists  aJid  of  SYSCOM  manager's  major  areas  of  in- 
terest. Therefore,  the  vsirious  report  categories  above  will  primairily  be 


sorted  and  printed  in  any  one  or  combination  of  the  following  groupings i 

• Chronological  Order 

• ACAT  Category  (l,  II,  III  or  IV) 

• Program/Acquisition  Manager  Organization 

• Program  Elements 

• T&E  Identification  Numbers  (TEINs) 

This  is  not  to  limit  the  flexibility  of  the  information  system  proposed. 

Unique  manager  interest  may  dictate  a number  of  other  sort  methods  as  well. 
Managers  may  desire  to  review  a report  of  T&E  efforts  listed  by  magnitude 
of  RDT&E  or  Procurement  funding,  or  a listing  of  all  programs  under  the 
sponsership  of  one  pairticular  CNO  Program  Sponsor.  The  T&E  information  sys- 
tem and  its  programmer  should  remain  innovative  sind  flexible  in  this  area. 

5.3-3  Preformat  Principle 

Because  it  is  possible  to  define  the  types  of  reports  required 
and  then  bound  their  format  with  a smaller  number  of  sort  combinations,  the 
composite  reports  can  be  stamdairdized.  The  computer  programmer  can  then  write 
specific  software  to  reproduce  these  standaird  formats  through  the  use  of  access 
codes  and  create  preformatted  report  outputs.  This  will  alleviate  the  ne- 
cessity to  reconstruct  the  form  of  the  report  each  time  the  data  is  to  be 
printed.  The  same  type  of  data  can  be  presented  in  any  number  of  different 
ways  depending  on  the  situation,  and  will  always  be  available  at  "the  push 
of  a button...".  The  preformatted  reports  can  include  a previously  designed 
cover  memorandum  so  that  a complete  package  of  information  and  forwarding 
documentation  can  be  created  in  one  operation.  Samples  of  this  cover  memo- 
randum are  provided  in  Appendices  D through  K as  attached  to  the  various 
sample  report  formats. 
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5. 3*^  Report  Numbering 


Since  there  will  be  a bounded  number  of  report  and  sort  combi- 
nations, each  type  of  report  can  then  be  described  by  certain  characteristics 
which  can  be  related  to  a numbering  system  that  will  facilitate  the  identi- 
fication, logging  2ind  distribution  of  the  information  packages.  The  illus- 
tration below  is  only  one  type  of  numbering  system  that  could  be  devised. 

The  finaJ.  system  must  be  tailored  to  the  output  formats  devised  within  indi- 
vidual computer  progirams. 


TYPE  OF  REPORT 

REPORT  SUBJECT 

PRIMARY  SORT 

SEItlAL  NUMBER 

1.  Box  Score 

1.  TEMPs 

1 . Chronological 

Local,  logging 

2.  Program  MS  List 

2.  OPEVAL 

2.  ACAT 

number  systan. 

3.  Major  MS  List 

3.  TECHEVAL 

3.  PM  Orgsuiization 

4.  Document  Update 

4.  ASU 

4.  Program  Element 

ETC 

5.  TEIN 

mm 

Using  the  system  above.  Report  Syrbol  321-08  would  indicate  that  the  report 
is  a listing  of  major  milestones  (in  this  case  - OPEVALs)  that  axe  printed 
in  chronological  order.  It  is  either  the  8th  report  sent  out  this  year,  the 
8th  variation  of  format  for  this  type  of  information  or  whatever  is  chosen 
for  the  serial  number  to  represent. 

5.3*5  Psuedo  Organization 

For  the  purposes  of  developing  sample  report  outputs,  fictitious 
programs,  milestone  dates  and  events,  organizations  and  other  data  were  used. 
The  psuedo  organization  is  the  Naval  Defense  Systems  Command  (NAVDEF)  and  its 
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project  management  organizations  are  referred  to  as  "PMXs".  The  NAVDEF  T4E 
Coordinator  is  "Code  OIT"  and  other  distribution  codes  and  administrative 
type  TEMP  data  have  been  manufactured  by  the  author  for  sample  purposes 
only. 

In  order  to  provide  continuity  in  the  numbers  contained  in  the 
sample  reports  and  a viable  cross-correlation  of  the  various  datai  the  fol- 
lowing matrix  of  baseline  program  information  was  developed  which  identifies 
the  fictitious  NAVDEF  components  and  the  number  of  programs  each  is  respons- 
ible for  within  each  AGAT  threshold i 


ORGANIZATION 

AGAT  I 

AGAT  II 

AGAT  III 

AGAT  IV 

TOTAL 

CODE  096 

1 

0 

1 

1 

3 

CODE  03 

1 

1 

2 

4 

8 

CODE  05 

1 

5 

4 

3 

13 

PMX  103 

1 

2 

3 

0 

6 

PMX  105 

3 

15 

9 

5 

32 

m 122 

2 

7 

1 

4 

14 

PMX  136 

2 

6 

0 

3 

11 

PMX  142 

5 

16 

16 

10 

47 

TOTAL 

16 

52 

36 

30 

13^ 

5.3*6  Box  Scores 

The  box  score  is  a status  display  device  designed  to  give  a 
quick  overview  of  statistics  in  a particular  airea  of  Interest.  Figure  5*^ 
illustrates  a staindard  box  score  format  - essentially  a matrix  which  matches 


1 CATEGORI  "A" 

CATEGOEY  "B" > 

TOTAL 

> 

f 

X X X X X X 

X X X X X X 

X 

X 

1 TOTAL 

X X X i X X 

X 

Figure  5.^  Sample  Box  Score 
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two  different  categories  of  information  and  allows  a calculation  of  sub- 
totals cind  a "grand  total"  of  the  numerical  data  presented.  The  advantages 
of  a box  score  stre  thati 

• It  is  easy  to  read. 

• It  will  provide  an  overall  picture  of 
status  without  having  to  know  many  details. 

• It  is  most  easily  accepted  by  the  "organization" 
since  is  displays  a minimum  of  "protected"  data. 

The  box  score  lends  itself  most  to  the  display  of  administrative 
progress  in  the  T&E  environment.  The  Systems  Command  Commander  is  basically 
interested  in  overviews  of  numbers  of  programs,  how  mauiy  plans  have  been 
written  or  need  to  be  written  auid  so  forth.  Because  individual  programs  are 
tailored  and  Involve  different  time  bases,  it  makes  no  sense  to  develop  box 
scores  where  no  commonality  exists  or  where  such  a tabulation  would  not  trans 
mit  usuable  data.  For  the  purposes  of  monitoring  T&E  status,  the  box  score 
in  this  system  will  be  used  to  display  the  following  categories  of  informa- 
tion » 

• Status  of  TEMPs 

• Status  of  TEPs 

• Status  of  Requests  for  ASU 

• Status  of  Requests  for  PASU 

Again,  this  choice  is  only  a staurtlng  point  for  this  baseline  information 
system.  There  atre  a number  of  ways  the  information  can  be  displayed  and  the 
choice  of  what  categories  to  display  in  the  matrix  should  be  determined  by 
individual  requirements.  In  addition,  box  scores  can  become  more  sophisti- 
cated (as  discussed  in  Section  6.2)  and  subsequent  generations  could  include 
such  items  ais  trend  analysis  and  progress  assessment  by  comparing  current 
box  score  information  with  previous  historical  data. 


59 


Samples  of  envisioned  Box  Score  reports  have  been  developed  to 
illustrate  the  status  of  fictitious  TEMPs,  TEPs,  ASU  and  PASU,  and  are  in- 
cluded as  Appendices  D and  E at  the  end  of  this  report. 

5. 3*7  ProKram  Milestone  Listings 

This  type  of  report  is  essentially  a print-out  of  part  or  all 
of  the  T&E  milestones  associated  with  a particular  program,  arranged  in  chrono- 
logical order,  but  formatted  in  the  following  categories  1 

I - OVERDUE  MILESTONES 

Focuses  management  attention  on  milestones 
scheduled  prior  to  the  report  date  that  have 
not  been  reported  as  complete,  auid  hence  have 
not  been  transfered  to  the  "Milestones  Completed" 
category  (VI  below) . 

II  - MILESTONES  RESCHEDULED  SINCE  LAST  REPORT 

Identifies  those  milestones  which  were  either 
previously  overdue  and  rescheduled,  originally 
erroneous,  or  chaunged  for  other  reasons.  This 
is  a one-time  listing  since  the  new  dates  will 
be  integrated  into  the  proper  section  in  the 
next  report. 

III  - MILESTONES  DUE  WITHIN  THE  NEXT  THREE  MONTHS 

Focuses  management  attention  on  eminent  events. 

Consists  of  a chronological  listing  of  events 
scheduled  for  the  following  three  months. 

IV  - MTT.ESTONES  DUE  IN  THE  FOLLOWING  TWELVE  MONTHS 

A longer  range  chronological  projection  of 
milestones  scheduled  for  the  following  yeax. 

This  twelve  month  listing  plus  III  above  covers 
a span  of  fifteen  months  total. 

V - MILESTONES  REMAINING 

Lists  outyear  projection  of  all  remaining  known 
or  estimated  milestone  dates  for  the  remainder 
of  the  program.  Prepared  as  an  enclosure  to  the 
report  because  of  its  potential  length. 

VI  - MILESTONES  COMPLETED 

Chronological  listing  of  milestones  previously 
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completed  (for  historical  purposes  euid  audit 
trail) . SimilaLTly  treated  as  am  enclosure  due 
to  its  potential  length  nearer  the  end  of  the 
program . 

This  type  of  report  is  not  intended  to  be  a regulatrly  prepared 
amd  distributed  document  since  one  would  be  required  for  each  of  the  50  to 
300  programs  under  the  purview  of  a Systems  Command.  It  does,  however,  pro- 
vide a good  representation  of  the  scope  of  a paurticulax  program,  amd  cam  be 
generated  on  demand  for  paurticulau:  mamagement  purposes.  A sample  of  this 
type  of  report  is  included  as  Appendix  F at  the  end  of  this  report. 

5.3.8  Ma.ior  Milestone  Listing 

More  functional  tham  the  preceding  Program  Milestone  Listing, 
the  Major  Milestone  Listing  provides  a consolidated  report  of  one  paurticulaur 
type  of  milestone  common  to  aill  programs.  For  instamce,  a "Report  of  TECHEVAL 
Milestones"  will  chronologically  list  every  TECHEVAL  currently  scheduled. 

This  type  of  report  has  a particulau:  appeal,  to  mamagement  since  it  is  much 
shorter,  more  easily  comprehended  amd  is  more  likely  to  amswer  specific  areas 
of  questions. 

With  a "sort  amd  print"  concept  of  computer  programming,  it  is 
conceivable  that  amy  number  or  type  of  milestone  can  be  formatted  into  this 
type  of  report  amd,  if  amy one  were  interested,  could  provide  reports  on 
due  dates  of  draft  or  final  test  plans,  test  reports,  program  plams,  contract 
awards  amd  the  like.  For  the  purposes  of  example,  six  representative  major 
milestone  categories  have  been  chosen  as  follows  1 

• DSARCs  • OPBVALS 

• NSARCS  • ASUs 

• TECHEVALS  • PASUs 
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This  report  also  lends  Itself  to  consolidating  categories  of 
information  which  are  the  subject  of  other  reports.  For  example,  the  Major 
Milestone  Listing  format  could  be  used  to  provide  commauid  data  on  all  over- 
due milestones  (Part  I of  the  Program  Milestone  Listings)  or  rescheduled 
milestones  (Part  II  of  the  same  report) . Samples  of  Major  Milestone  Listing 
reports  are  provided  as  Appendices  G through  J at  the  end  of  the  report. 
These  samples  were  abbreviated  in  order  to  illustrate  format  vice  detailed 
content . 

5.3*9  Document  Uixiate  Status 

The  last  format  in  the  initial  version  of  the  T&E  Information 
system  deals  with  keeping  track  of  the  requirements  to  update  a number  of 
documents  at  vaurious  times  throughout  the  program,  either  on  an  annual  basis 
or  at  other  prescribed  times.  For  example,  TEMPs  are  to  be  updated  annually 
and,  if  this  fact  alone  were  enough,  the  updating  ixrocess  would  be  easy  to 
track.  However,  TEMPs  must  also  be  updated  (in  accordance  with  OPNAVINST 
3960.10)  at  least  two  months  prior  to  a major  decision  point  (l.e.  DSARC) . 

If  a scheduled  TEMP  update  slips,  its  next  annual  update  is  impacted.  If  the 
annual  update  date  is  close  enough  to  a major  decision,  only  one  update 
might  be  required.  This  requirement  can  be  manually  tracked  for  a small 
number  of  programs  but,  if  one  considers  the  need  to  report  the  current 
status  of  300  programs  residing  in  different  program  offices,  the  advantages 
of  this  report  format  become  obvious. 

The  dates  provided  by  this  report  are  computed  by  the  computer 
by  calculating  time  differences  based  on  the  current  report  date  entered. 
That  is,  comparing  (DSARC  - 2 months)  and  (Last  TEMP  Update  + 12N)  where 
N=l,2,3  etc  years,  the  next  required  milestone  can  be  determined. 
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A sample  report  is  provided  as  Appendix  K at  the  end  of  this 
report,  illustrating  the  status  of  TEMP/TEP  required  updates.  Again,  this 
sample  is  abbreviated  to  illustrate  only  one  format.  The  report  can  zJ.so 
be  used  to  provide  status  on  other  document  update  requirements  such  asi 

• DCPs  and  NDCPs 

• Program  Management  Pleuis 

• System  Engineering  Master  Plans 

• ILS  Plauuiing 

Etc 


6.0  SUMMARY 

6.1  GENimL 

The  preceding  section  has  just  described  four  types  of  possible  reports 
that  could  be  generated  by  a computerized  information  system  based  on  a 
structured  consolidation  of  data  contained  in  existing  Test  and  Evailuation 
Maister  Plaoa  (TEMP)  docmentation  and  other  acquisition  sources.  The  require- 
ment for  such  a system  is  derived  from  the  fact  that,  although  individual 
Project  and  Acquisition  Managers  adequately  mainage  the  T&E  efforts  of  their 
specific  programs,  very  little  capability  now  exists  within  the  Naval  Mat- 
erial Gommeuid  to  assess  the  total  picture  - that  is,  to  relate  on  a reliable 
and  regular  basis  the  status  of  key  T&E  milestones  and  the  preparation,  sub- 
mission and  approval  of  mandatory  T&E  documentation.  T&E  Coordinators  have 
been  established  to  provide  such  a conmand  function,  but  no  administrative 
system  yet  exists  that  will  allow  that  coordinator  to  easily  display  such 
data  in  flexible  and  diverse  report  formats. 

It  would  be  easier  if  we  were  able  to  bypass  this  coordination  function 
and  rely  solely  on  each  Project  Manager  to  provide  an  overview  of  his  pro- 
'gpam's  T&E  status  to  senior  management.  However,  in  today's  current  environ- 

X 

ment  of  acquisition  policy,  the  Chief  of  Naval  Material  and  Systems  Command- 
ers now  need  a more  Integrated,  consistent  and  regulaurly  scheduled  report 
of  T&E  status  in  order  to  recognize  the  significance  of  the  T&E  effort  as  a 
prerequisite  for  successful  DSARC  decisions,  and  to  respond  (in  a corporate 
sense)  to  the  increstsing  visibility  and  importance  being  placed  on  test  and 
evaluation  events  by  the  CNO,  SECNAV,  OSD  and  Congress. 
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6.2  FUTURE  SOPHISTICATION 
6.2.1  General 


The  Information  system  described  in  this  paper  is  a simple  one,  | 

and  is  not  designed  to  provide  any  profound  revelations  or  create  handy  i 

decision  tools.  It  is  merely  a concept  - a meams  to  display  T&E  data  in  a I 

usuable  form,  to  hopefully  highlight  critical  events,  and  provide  an  over-  | 

all  identity  to  the  network  of  inter-related  information  which  would  other-  j 

wise  remain  in  saife  drawers  filed  neatly  away  in  TEMP  folders.  The  proposed  j 

system  is  a starting  point  and,  as  it  becomes  operational,  can  be  built-upon  | 

in  a modular  fashion  and  expanded  into  a more  complex  euid  active  management  j 

3 

tool.  The  examples  contained  in  the  following  subsections  provide  some  ^ 

idesus  for  the  future  sophistication  of  this  system, 

6.2.2  Trend  Capability 

As  the  data  is  updated,  the  computer  can  be  programmed  to  store 
previous  information  in  accordance  with  a structured  time  base.  The  current 
data  can  then  be  compared  with  previous  data  auid  a trend  ceui  be  calculated 
Indicating  increases,  decreases,  no  chauiges  or  whatever  indices  of  progress 
is  desired.  This  trend  auiadysis  can  be  generated  on  a short-term  basis  by 
compaxlng  only  the  data  from  the  last  report,  or  on  a longer-term  basis 
by  compaxlng  cumulative  data  or  historical  trend  Indices. 

6.2.3  Graphical  Capability 

Once  a historical  data  base  is  achieved,  the  computer  can  be 
programmed  to  provide  a graphical  output;  a mode  which  can  automatic*J.ly 
adjust  the  range  of  graph  scales  in  accordance  with  the  scope  of  data  to 
be  plotted  and,  in  some  cases,  may  even  be  capable  of  drawing  the  plot  in 
three  or  four  colors.  The  graphical  capability  is  most  effective  in 
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displaying  the  cumulative  data  contained  in  the  Box  Score  Report.  Figure 
6.1  illustrates  an  example  of  a potential  TEMP  Box  Score  graph. 
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Figure  6.1  TEMP  Box  Score  Status 
6.2.4  Chart  Capability 

As  in  the  above,  the  computer  can  also  be  programmed  to  provide 
milestone  date  information  on  a chart  display  in  addition  to  a chronological 
list.  Figure  6.2  provides  a sample  of  how  such  a chart  might  look. 
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Figure  6.2  Milestone  Chart  Display 
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Due  to  space  considerations  the  time  scale  is  in  fiscal  year  quarters  and 
specific  dates  are  not  indicated  (although  the  plot  could  be  scaled  in 
months  or  days  for  a short-term  display) . An  jidditional  coding  scheme  on 
the  input  data  would  be  required  to  indicate  whether  a date  was  a "start" 
date  or  a " complete"  date.  A more  sophisticated  chart  could  be  programmed 
to  display  both  current  and  historical  data  for  analysis  purposes. 

6.2.5  Milestone  Interactions 

Once  a set  of  T&E  milestones  has  been  chronologically  arranged 
for  each  program,  it  is  possible  to  determine  and  store  the  time  intervals 
between  selected  key  milestones  (i.e.,  establish  a time  relationship  be- 
tween each) . If  a change  in  a milestone  date  occurs , it  will  disturb  the 
previous  order  of  intervals,  and  the  computer  can  reccmpute  the  network 
based  on  the  previous  relationships  and  propose  a resultant  milestone  re- 
structure. This  manipulation  would  be  similax  to  the  Performance  Evaluation 
auid  Review  Technique/CJrltlcal.  Path  Method  (PERT/GPM)  used  in  other  computer 
scheduling  evaluations. 

The  danger  with  this  method  is  that,  in  mauiy  cases,  T&E  mile- 
stones slip  but  other  dates  must  remain  sacred.  Rescheduling  of  the  entire 
sequence  is  not  done  each  and  every  time  auid  certain  Impact  considerations 
must  be  weighed  in  each  milestone  delay  situation.  However,  the  ccmputer 
caul  provide  a relative  indication  of  critical  paths  auid  impacts  to  be  con- 
sidered during  the  humaui  analysis  phase. 

6.2.6  Acquisition  Milestones 

The  data  bause  caui  be  expanded  to  Include  more  or  all  of  the 
total  aoquisition  process  milestones.  Since  the  method  of  prepeLrlng  TEMPs 
differs  from  program  to  program  and  the  term  "Major  Milestones"  in  the  TEMP 
schedule  format  is  subject  to  wide  Interpretation,  the  input  data  described 


in  Section  5*2  does,  in  fact,  include  more  milestone  dates  thsui  normally 
attributed  to  the  pure  T&E  process.  This  was  done  not  only  to  provide  a 
measure  of  flexibility  in  translating  existing  TEMP  data  into  input  data, 
but  also  to  establish  a more  detailed  frame  of  reference  so  that  the  T&E 
milestones  can  be  related  to  the  total  time  framework  of  the  acquisition 
process.  The  adding  of  additional  milestones  will  enable  the  system  to 
display  a total  chronological  listing  of  acquisition  events  for  each  prog- 
ram. There  is  a point  of  departure  in  this  concept,  however.  The  list  of 
potential  milestones  is  never  ending  and  could  even  extend  into  the  depths 
of  the  production  process  on  the  contractor's  plant  floor.  A certain  meas- 
ure of  common  sense  is  required  when  establishing  the  scope  of  such  a data 
expansion . 

6.2.7  Organizational  Utili2ation 

The  proposed  information  system,  although  designed  for  use  by 
the  Gcanmetfid  T&E  Coordinator  to  provide  an  across-the-board  assessment  of 
commauid  T&E  status,  it  could  also  be  utilized  in  more  detail  within  the 
Project/Acquisition  Manager's  orgsuiization  and,  in  a more  general  sense, 
could  be  applied  at  the  NAVMAT  Staff  level.  That  is,  those  key  milestones 
which  Identify  major  events  such  as  ASU,  OPEVAL,  TECHEVAL  euid  DSARGs,  and 
the  administrative  data  such  am  TEMP/TEP  documentation  status.  The  inclusion 
of  any  more  detailed  technical  data  would  not  be  feasible  at  this  higher 
command  level  since  these  milestones  atre  less  critical,  more  subject  to 
fluctuation  and  within  the  PM's  responsibility  to  manage. 
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6.3  IMPLEMENTATION 

There  has  been  little  reference  to  implementation  within  this  paper 
other  than  providing  a partial  plan  of  action  in  Section  4.3.  Each  command 
must  tailor  the  system  and  its  implementation  to  its  own  unique  organization 
and  internal  command  requirements.  It  is  important  to  emphasize  that,  what- 
ever implementing  procedures  are  developed,  they  should  be  fully  integrated 
with  existing  T&E  directives  and  current  methods  of  doing  business.  The  old 
auiage  of  "Garbsige  in-Gatrbage  out"  will  soon  overtsdce  the  system  if  care  is 
not  taken  to  avoid  the  situation.  This  is  particularly  true  when  the  system 
is  first  implemented  because  there  will  be  high  resistance  to  any  change 
and  to  perceived  increases  in  workloaid. 

The  maintenance  and  use  of  the  system  should  become  part  of  the  over- 
al.1  command  T&E  process.  Currently,  directives  define  specific  actions  to 
be  taken  in  prosecuting  T&E  related  efforts,  even  to  the  point  of  listing 
the  sequence  of  intemgj.  "chop"  codes  and  approve!  points.  Unless  these 
procedures  also  include  the  actions  required  to  insure  that  the  new  data 
is  properly  inputted  to  the  computer,  the  system  will  be  bypassed  and  soon 
lose  credibility  due  to  stale  information.  It  is  strongly  recommended  that 
once  a decision  has  been  made  to  develop  auid  implement  the  data  base,  ex- 
isting T&E,  TEMP/TEP  and  ASU  directives  be  reissued  concurrently  with  sys- 
tem implementation  to  insure  that  the  proper  data  input  checks  and  balances 
have  been  provided. 


6.4  THE  END 

The  introduction  stated  that  this  test  cUid  evaduation  information 
system  fulfills  a particular  need.  It  may  not  be  directly  applicable  in 
other  command  requirements,  but  it  is  hoped  that  some  of  the  concepts  pro- 
posed maty  provide  a glimmer  of  thought  and  maty  motivate  other  individuads 
to  develop  their  own  tailored  system  to  solve  their  own  problem.  If  this 
happens,  this  paper  has  fulfilled  its  function. 
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APPENDIX  A 

TEST  AND  EVALUATION  MASTER  PLAN  (TEMP) 

BASIC  FORMAT 

PART  I.  ADMINISTRATIVE  INFORMATION 

PART  II.  DESCRIPTION 

1.  System  Description  and  Mission 

2.  Critical  T<ScE  Issues 

3.  Objectives  auid  Thresholds 

4.  Required  Technical  Characteristics 

5.  Required  Operational  Characteristics 

6.  Environmental  Impact  Assessment  of  T&E 

PART  III.  INTEGRATED  SCHEDULE 

1.  Program  Milestones 

2.  Pertinent  T&E  Data 

3.  Major  Resource  Availability  Requirements 

4.  Key  Dates  for  Test  Plauis  and  Reports 

PART  IV . DT&E  OUTLINE 

1.  DT&E  to  Date 

2.  Future  DT&E 

a.  Equipment  Description 

b.  DT&E  Objectives 

c.  DT&E  Events/Scope  of  Testing/Basic  Scenarios 

d.  Quantifiable  Scope  of  Effort 

3.  Critical  T&E  Items 

PART  V.  OT&E  OUTLINE 
PART  VI.  PAT&E  OUTLINE 

PART  VII.  RESOURCE  SUMMARY 

1.  Test  Articles 

2.  Fleet  RDT&E  Support 

3.  Test  Sites/Ranges 
<*.  Targets 

>.  special  Instruaentation 


A 1 


L' 


6.  Support  Equipment 

7.  Installation/Removal  Requirements 

8.  Expendables 

9.  Logistics  Support 

10.  Personnel 

11.  Personnel  Training 

12.  Planned  Travel 


X 13.  other  (as  necessary) 

< 

i PART  VIII.  REFERENCES 


APPENDIX  B 


CONSOLIDATSD  T&E  COORDINATOR  FUNCTIONS 


1.  Develops  GommaJid  Test  auid  Svaduatlon  policies  aJid  procedxires 
applicable  to  the  development  £Uid  acquisition  process. 

2.  Ensures  the  Gcsnmand's  compliance  with  higher  level  requirements. 

3.  Programs  ajid  administers  those  T&E  support  funds  under  T&E 
coordinator  cognizance. 

4.  Reviews  the  Gommeuid's  T&E  M2ister  Plauis  (TEMPs)  euid  T&E  Plans  (TEPs) 
for  compliance  with  established  criteria. 

5.  Monitors  the  status  of  Command  T&E  programs  and  provides  status 
reports  to  the  Command  on  T&E  issues. 

6.  Maintains  liaison  with  GOMOPTEVFOR  and  other  Gommauids  to  establish 
Command  priorities  for  Fleet  services  to  assure  coordinated  SYSCOM 
DT&E  and  OT&E. 

7.  Reviews  all  requests  fron  SYSCOM  to  GNO  for  T&E  project  aissignments . 

8.  Reviews  "equipment  readiness"  reports  for  Command  projects  assigned 
to  GOMOPTEVFOR  for  test  and  evaluation. 

9.  Reviews  for  the  Command  requests  for  Approval  for  Service  Use  for 
compliance  with  established  T&E  criteria. 

10.  Provides  guidance  on  matters  of  Gcanmand  T&E  policy  to  field  activities 
supporting  T&E  programs. 

11.  Provides  guidance  and  assistance  in  planning  for  the  use  of  existing 
or  new  Land  Based  Test  Sites  and  other  test  facilities. 

12.  Manages  the  development  and  supporting  implementation  of  the  SYSCOM 
portion  of  the  Total  Ship  Test  Program. 

13.  Provides  assistance  to  Program/Acquisition  Managers  in  the  following 
areas  t 


a.  Preparing  T&E  portions  of  DSARG,  NSARC,  and  CEB  General  Board 
presentations  and  similiar  reviews. 

b.  Preparing  and  updating  T&E  documentation  such  eis  TEMPs,  Test 
Plans,  Test  Reports,  the  T&E  portion  of  DCPs,  PMs,  NDCPs,  DPs,  APPs, 
and  contract  specifications. 

c.  Planning,  preparing  and  justifying  T&E  budgets. 

d.  Preparing  and  subnltting  the  T&E  Section  of  Congressional  Data 
Sheets  and  Program  Element  Descriptive  Summaries. 
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APPEiroiX  c 


SAMPLE  IMPUT  DATA  WORKSHEET 

Refer  to  Parts  I aind.  Ill  of  Test  and  Evaluation  Master  Plan  (TEMP) . 
Other  data  may  be  required  from  other  sources. 

Enter  dates  as  follows i 2 digit  day  - 3 letter  month  - 2 digit  year 

(03  NOV  78) 

Indicate  missing  data  by  providing  the  following  on  the  blanks i 
"NA"  - Not  Applicable 
"TBP"  - To  be  Provided 

I - ADMINISTRATIVE  DATA  (TEMP  Part  I) 

1.  TEINi  

2.  Full  Program  Title « 

3.  Program  Elanent  Numberi  

4.  Project  Numberi  

5.  AGATi  _ _ _ 

6.  RDT&E  Fund  Level  (Mil)  i . 

7.  Procurement  Fund  Level  (Mil)  i . 

8.  GNO  Program  Sponsor i 

Name  t _ 

Code  1 

Tell  ____  

9.  Project  Manager I 

Name  i _ 

Code  I 

Tell  ________________ 

10.  Acquisition  Manager i 

Name  i _ 

Code  I 

Tell  

11.  Test  Director I 

Nsunei 
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Code  I 

Tell  ___  ___  

12.  Science  & Technology  Objectives  Document  (S&TO) 

Number!  

Date  I 

13.  Operational  Requirement  (OR) 

Numberi  

Date  t 

14.  Development  Proposal  (DP) 

Numberi  

Date  I 

15.  Program  Memorandum  (PM) 

Numberi  

Date  I 

16.  Navy  Decision  Coordinating  Paper  (NDCP) 

Numberi  

Date  I 

17.  Decision  Coordinating  Paper  (DCP) 

Numberi  

Date  I 

18.  Test  and  Evaluation  Master  Plan  (TEMP)  (Or  TEP) 

Date  of  Original  Approval  1 _ 

Date  of  Latest  Revision  1 _ 

II  - INTEGRATED  SCHEDULE  DATA  (TEMP  Part  III) 

A.  MAJOR  MILESTONES 

1.  Milestone  (Program  Initiation)!  

2.  NSARCS 

a.  NSARC  I I _ _ _ _ _ _ 

b.  NSARC  II  • __  _ _ _ _ 

c.  NSARC  III  « _ _ __  _ __ 

d.  NSARC  Illai  _ _ __  _ _ 

e.  NSARC  nib  I 


C-2 


3.  DSARCS 

a.  DSARC  I I 

b.  DSARC  II  • _ _ _____  _ 

c.  DSARC  III  I _ _ _ _ _ 

d.  DSARC  Illai  _ __  _ _ _ _ 

e.  DSARC  Illbi  ___  _ _ __ 

4.  Technical  Reviews 

a>  System  Requirements  Review  (SRR) i 

\ b.  System  Design  Review  (SDR) i 

c.  Preliminary  Design  Review  (PDR) t 

d.  Critical  Design  Review  (CDR) i 

e.  FimctionsLl  Config  Audit  (FGA)  i 

f.  Physical  Config  Audit  (PCA) « 

, g.  Prod  Readiness  Review  (PRR) t 

5.  Major  Equipment  Installations 

a.  Prototype  1 

b.  Production  

c.  Other!  

6.  Certification  of  Readiness  for  OPEVAL 

a.  Requested!  __  __ 

b.  Granted!  

7.  Provisional  Approval  for  Service  Use  (PASU) 

a.  Requested!  

b.  Granted! 


b.  System  Engineering  Management  Flan  (SEMP) 

Start  I 

Draft  Duel  ___  _ . 

Final  Duet  

c.  Integrated  Logistics  Support  Flan  (ILSF) 

Start  I 

Draft  Duel  

Final  Duel  

d.  Source  Selection  Flan  (SSF) 

Start  I 

Draft  Duel  

Final  Duel  

e.  Development  Specification 

Start  I 

Draft  Duel  

Final  Duel  

f.  System  Specification 

Starti  

Draft  Duel  

Final  Duel  

g.  Product  Specification 

Start  I 

Draft  Duel  

FlneJ.  Duel  

h.  Material  and  Process  Specifications 

Starti  

Draft  Duel  __  

Final  Duel  

i.  Project  Summary  Work  Breakdown  Structure  (WBS) 

Start  I 

Draft  Duel  __  _ __  __  __  __ 

Final  Duel 
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j.  Contract  Work  Breakdown  Structure  (CWBS) 

Starti  

Draft  Duel  ___  

Final  Duel  

k.  Request  for  Proposal  (Ciorrent) 

Starti  

Draft  Duel  _ __  _ 

Final  Duel  

10 . Initial  Operational  Capability  (lOC) i 

11 . Full  Operational  Capability  (FOC) i 

12.  Other  Major  Milestones 


a.  I 

b.  I 

c.  I 

d.  I 


B . Contract  Dates 

1 .  Contract  Awards 

a.  Conceptual! 

b.  D«n-Vali 

c.  Full  Scale  Eng  Devi 

d.  Pilot  Production  Model i 

e . Production i 

C.  DCP  or  NDCP 

1.  DCP  Updates  1 


2.  NDCP  Updates  1 


D.  TEMP 

1.  TEMP  Updates  I 
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E.  Test  Articles 


1.  Brassboardi 

2 . ADM  Delivery i 

3.  ADM  Tests 
Starts 
Complete  s 

4.  EDM  Delivery s 

5.  EDM  Test 
Starts 
Complete  s 

F.  DT&E 

1.  DT-I  Lab 
Starts 
Complete  s 

2.  DT-I  Shore 
Stjirts 
Complete  s 

3.  DT-I  At  Sea 
Starts 
Complete  s 

4.  DT-IIa 
Starts 
Complete  s 

5.  DT-IIb 
Starts 
Complete  s 


DT-IIIa 
Starts 
Ccmiplete  s 


?.  DT-IIIb  (TEGHEVAL) 


Start  I 


Complete i 


G.  OT&E 


1.  OT-I 


Stsurti 


Ccmiplete  i 


2.  OT-IIa 


Start! 


Complete  t 


3.  OT-IIb 


Start! 


Complete ! 


OT-IIIa 


Start! 


Complete ! 


5.  OT-IIIb  (OPEVAL) 


Start! 


Complete ! 


6,  OT-IV  (As  Required) 


Start! 


Complete ! 


7.  OT-V  (As  Required) 


Start! 


Complete ! 


H.  PAT&E 


1.  PAT&E  (As  Required) 


Start! 


Complete i 


•m 


I.  DT&E  TEST  PLAN 

1.  DT-I 

a.  Staxti  

b.  Draft  Duet  

c.  Final  Duel  

2.  DT-II 

a.  Staurti  

b.  Draft  Duel  

c.  Final  Duel  

3.  DT-IIIa 

a.  Start  I 

b.  Draft  Duel  

c.  Final  Duel  

4.  DT-IIIb  (TECHEVAL) 

a.  Start  I 

b.  Draft  Duel  

c.  FineJ.  Duel  

J.  OT&E  TEST  PLAN 

1.  OT-I 

a.  Start  I 

b.  Draft  Duel  

c.  Final  Duel  

2.  OT-II 

a.  Start  I 

b.  Draft  Duel  

c.  Final  Duel  

3.  OT-IIIa 

a.  Start I 

b.  Draft  Duel  __ 

c.  Final  Duel 


C-8 


4.  OT-IIIb  (OPEVAL) 

a.  Steurti 

b.  Draft  Duel 

c.  Final  Duel 

DT&E  TEST  REPORT 

1.  DT-I 

a.  Start  I 

b.  Draft  Duel 

c.  Final  Duel 

2.  DT-II 

a.  Start  I 

b.  Draft  Duel 

c.  Final  Duel 

3.  DT-IIIa 

a.  Starti 

b.  Draft  Duet 

c.  Final  Duel 

4.  DT-IIIb  (TECHEVAL) 

a.  Starti 

b.  Draft  Duel 

c.  Final  Duel 

OT&E  TEST  REPORT 

1.  OT-I 

a.  SteLTti 

b.  Draft  Duel 

c.  Final  Duel 

2.  OT-II 

a.  Starti 

b.  Draft  Duel 

c.  Final  Duel 


APPENDIX  D 


I uCTutitrf  i97o  < 


ffAVt>fcf  r<ire  -Gt-Kj kt)  1 y< ATJh  (vJuDt:  ulf) 

TG«  UlSTr^lBUTIQN  LIST 

TcMB/TcB  dGX  SCJht  rGk  ockrtMeck  19/d 

kth*  (A)  NAVDtriNST  j9o0.41A 


h.MCL*  (1)  ftMF  oTArUS  BY  ACA'i 

(2)  i'tMP/TfcP  STATUS  BY  UkoAiMUATIoN 

1.  IN  ACCGhUANCt  AiTrl  kfcrckfcNGt  (A),  Trth  ruLLGAlNG  SUV.MAkY 
BGX  SC'JktS  Akt  FRGVIDeO,  UhklJTlNG  Xrih  SlATUb  Jr  TumF  ANC 
Tfcii  ^TitFAkAT TUN  AS  uF-  fkfc  eND  Jr  SekIfeMBtk  197B.  rUkTBfck 
BRcAK-GjaNS  dY  ACAI  ANG  by  GhGANlZATlJN  Akc  rkJy/ICcJ  BY 
ti'iCLGSUhcS  (1)  AND  (2)  ktSFtC f I VtLY . (Nulc*  FtkCbNT  hUUALS 
APPHQVhD/NK  FROGS). 

TnMF  STATUS  ( ACATS  I,  11,  AND  111) 


CGUt Nk  FROGS  kH  I fT NGf_kRX^^  AFFk  J YtD  FtRJcNT 

096  2 I I I 50.0 

03  43  11  25.0 

05  10  6 4 4 40.0 

-WOUIOG  - 6 4 2 2 3J.3 

F'lAX  105  27  14  13  o 29.9 

FMX  122-10  7 3 7 /O.O 

FMX  136  8 2 o 1 12.5 

FMX  142  37  21  16  lb  40.5 

TJTA^- 404  -58-  46 .39 37.5 


TcF  STATUS 

(ACAT  IV) 

OuDt: 

Nk  FkuGS 

kRITTtN 

NOT  kkITTsN 

AFFRGVcD 

FtkOcNT 

096 

1 

1 

0 

1 

100.0 

Vo 

4 

2 

2 

1 

25.0 

05 

3 

3 

0 

2 

06 .6 

FMX  103 

0 

0 

0 

0 

00.0 

FMX  105 

5 

0 

5 

0 

00.0 

PMX  1 22 

4 

2 

2 

2 

50.  C 

FMX  136 

3 

2 

1 

2 

66.6 

FMX  1 42 

10 

6 

4 

4 

40.0 

TUrAL 

7 

30 

1 6 

14 

1 2 

40.0 

D-1 


» 


TtM'P  STATUS  BY  ACAT 


ACAf  I 


CQUt:  NdX-jQLiiXXXcU  AHt^kUM-tiD  Ht.iAit:dX 

096  1 I 0 1 100.0 

OJ  1101  lOO.O 

05  I 1 0 1 100.0 

PMX  103  I I 01  100.0 

HMX  105  3 2 I I 33.3 

2 1 Y 1 50.0 

PMX  136  2 2 0 I 50.0 

PMX  142  5 4 1 3 60.0 

TOTAL  16  13  J 10  — 62.5 


ACAT  II 


COOt 

NH  PHOOS 

AiinsN 

NJr  WHifTcN 

APPPGVcD 

PcHCcNT 

096 

0 

. . ._Q. 

0 

.Jl  _ . 

_oo^a 

03 

1 

1 

0 

0 

00.0 

05 

5 

4 

1 

2 

40.0 

PMX  103 

2 

1 

1 

1 

50.0 

PMX  105 

15 

10 

5 

6 

40.0 

PMX  122 

7 

6 

1 

6 

35.7 

PMX  136 

6 

0 

6 

0 

00.0 

PMX  142 

16 

14 

2 

10 

62.5 

TOTAL 

52 

36 

16 

25 

4d.8 

ACAf  III 

CO^ NP  PROPS 

AR  I I'TbA  NOT  WR 1 1 TcN 

APphOVtD 

PhRCtNT 

uyo 

1 

u 

1 

0 

00 . 0 

03 

2 

1 

1 

0 

dO.O 

05 

4 

1 

3 

1 

25.0 

PMX  103 

3 

2 

1 

0 

00.0 

PMX  105 

9 

2 

7 

1 

1 1 • 1 

PMX  122 

1 

0 

1 

0 

00.0 

PMH  1 36 

0 

0 

0 

0 

00 . 0 

PMX  142 

16 

3 

13 

2 

12.5 

TOIAL 

36 

9 

27 

4 

11.1 

ENCLUSUHt  ( 1 ) 


TtMP/TtP  STATUb  BY 

organization 

n iPt. 

ACAT 

NR 

PROGS 

WRiTTtN  NOT 

written 

APPRUVhD 

PhRCENT 

I 

1 

) 

0 

1 

ICX).0 

11 

0 

0 

0 

0 

00.0 

111 

I 

0 

1 

0 

00.0 

IV 

1 

1 

0 

1 

100. c 

TOTAL 

0 

2 

1 

2 

60.6 

cope  03 

AC^ 

Hk. 

PROGS 

uRITTfcN  NOT 

written 

approved 

PhRCENT 

1 

I 

1 

0 

1 

100.0 

II 

1 

j 

0 

0 

fXi  o 

III 

2 

1 

1 

w 

0 

00.0 

IV 

4 

3 

p 

1 

im 

1 

TUTAL 

8 

- - 

-3 

■ i - 

2S.0 

COUe  05 

ACAT 

NR  PROGS 

rRITTeN 

NOT  WRlTThN 

approved 

PhRChNT 

I 

t t 

1 

c. 

1 

0 

1 

100.0 

I i 

5 

4 

1 

2 

40.0 

III 

T U 

4 

1 

3 

1 

25.0 

I ¥ 

TOT A I 

3 

3 

0 

A 

2 

66.6 

t 

PMX_J.03 


ACAT 

NR  PROGS 

RfllTThN 

NOT  WRll  Th'N 

approved 

percent 

I 

1 

1 

0 

1 

100.0 

11 

2 

1 

1 

1 

50.0 

III 

3 

2 

1 

0 

00.0 

IV 

0 

0 

0 

0 

00.0 

TDTAr 

6 

4 

2 

2 

■ 33.3 

bNCLUbUKb  U) 
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I 


YtMP/Tt-P  STATUS  BY  JHGANlZATlUN  (CUNT) 


£.!4JL1Q5 


ACAT 

Wk  PROGS 

kRITlbN 

NOT  WRITltN 

APPkJVfcD 

PhHCtM 

I 

J 

2 

1 

1 

33.3 

II 

15 

10 

5 

6 

40.0 

III 

9 

2 

7 

1 

11.1 

IV 

5 

0 

5 

0 

00.0 

TOTAL 

32 

14 

18 

d 

25.0 

PMX  122 


ACAT 

NH  PROGS 

HRITTtN 

NOT  AHITTcN 

APPHOVtD 

PtRCtNT 

1 

2 

1 

1 

1 

50.0 

II 

7 

6 

1 

6 

85.7 

III 

1 

0 

1 

0 

00.0 

IV 

4 

2 

2 

2 

50.0 

.lOlAL. 

-14 

- 9 

5 

9 

64.3 

PMX  136 

A£AI  NH  PROPS  rifilllfeN  tiOI-ildilltti  APPkQVtD  Pfa^CtNT 

1 2201  50.0 

U 6 0 6 0 00.0 

III  0 0 0 0 00.0 

IV  3 2 1 2 66.6 

TOTAL  -44 4 7 4 27.3 


PMX  142 

AQAX  iili_£aQ5£S  aHillfcN  A£tiiiQVtD  P_LrfCt:NT 

I 5413  60.0 

II  16  14  2 10  62.5 

III  16  3 13  2 12.5 

IV  10  6 4 4 40.0 

TOTAL  - 44-  20  19  40.4 


ENOLOSUPb  (2) 


D-5 


APPENDIX  E 


r 

01  JULY  1978 

1? 

TO* 

DISTRIBUTION  LIST 

SUBJ* 

ASU/PASU  BOX  SCORE  pOR  JUNc  1978 

REP* 

(A)  NAVDEPINST  3960.41  A 

ENCL* 

( 1 ) ASU  STATUS  BY  ACAT 

<2>  ASU  STATUS  BY  ORGANIZATION 

(3)  PASU  STATUS  BY  ACAT 

(4)  PASU  STATUS  BY  ORGANIZATION 

1.'  i-fi  AeCt^ki^ANCE  /riTH  kfc^t:f^tNe^  (A),  TricMJLLQnlfiO  SUMMARY 
BOX  bCuHtS  ARE  PROVIDED,  DcPICIiNO  THE  STATUS  Of  ASU  AI'*D  PASU 
APPROVALS  AS  OP  THE  END  OF  JUNE  I97b.  PURTHtR  BREAKDOWNS  QP 
ASU  AND  PASU  BY  ACAT  AND  OROANIZATIUN  Ahc  PROVIDED  BY  ENCL 
(J)  THROUDH  (4).  (NOTE*  PERCENT  EQUALS  DRANTED/CY  ASU  SKED). 


AStl-STA'ftfS  HACATS  I-IVT 


CjDc 

NR  PROGS 

CY_A^_SKtD 

REQUhSTED 

GRANTED 

PcRCEN  1' 

096 

3 

0 

0 

0 

00.0 

03 

8 

2 

1 

0 

00.0 

05 

13 

2 

2" 

1 

50.0 

PMX  103 

6 

1 

0 

0 

00.0 

PMX  105 

32 

4 

3 

2 

50.0 

PMX  122 

14 

5 

4 

3 

60.0 

PMX  136 

1 1 

2 

2 

1 

50.0 

PMX  142 

• 47 

9 

9 

4 

44. 4_ 

TOTAL 

134 

25 

21 

1 1 

44.0 

PASU  STATUS  (ACAT3  I-IV) 


NR  PROGS 

iiX-ASiU_sI£a 

grANTcD 

096 

3 

1 

1 

1 

100.0 

03 

8 

0 

0 

0 

00.0 

05 

13 

0 

0 

0 

00.0 

PMX  103 

6 

0 

0 

0 

00.0 

PMX  105 

32 

2 

1 

1 

5U.0 

PMX  122 

14 

0 

0 

0 

00.0 

PMX  132 

1 1 

1 

1 

0 

00.0 

PMX  142 

47 

2 

1 

1 

50.0 

TOTAL 

134 

6 

4 

3 

50.0 

E-1 


PMX  103 
HMX  105 
PMX  122 
PiyW  4-36 
PMX  142 

COPY  TGi 

01 

OWlrlLfi 


M.  SWI^T 


ASU  STAIUS  bY  ACAT 


ACAT  1 

cjDc  £x_Asy_s]iHi2  taa^tNi 

096  10  0 0 00.0 

03^  f » 1 O 00.0 

05  ill!  100.0 

PMX  103  1 — 0 0 — 0 00.0 

PMX  105  3 2 2 1 50.0 

PMX  122  2 2 2 2 100.0 

PMX  136  2 1 1 1 100.0 

■PMX-+42 5 3 2 1—-  33^3 

TOTAL  16  10  9 6 60.0 


ACAI_il 

tTC  ... 


ACAI_1U 
fcfC  v.i . 


ACAl-iii 


ETC 


ASU  STATUS  BY  UHGANlZATlUN 


.Qaut_Q26 


ACAf 

NH  PROGS 

CY  ASU  SKfaO 

requested 

ORANTE'D 

PcRCtNI  ' 

I 

1 

Q. 

0 

Q 

CX).D_ 

11 

0 

0 

0 

0 

00.0 

III 

1 

0 

0 

0 

00.0 

IV 

1 

0 

0 

0 

OU.O 

TOTAL 

3 

0 

0 

0 

00.0 

acat 

NH  PfiOOS 

CY  ASU  SKED 

REQUESTED 

GRANTED 

H4HCENX 

i 

1 

I 

i 

Q 

- 

II 

1 

] 

T 

0 

0 

wV  • V/ 

00.0 

III 

n 

n 

rv 

r\(\  r\ 

£ 

\j 

\J 

uu  • u 

IV 

4 

0 

0 

0 

00.0 

TOTAL 

8 

2 

1 

0 

00.0 

CJUt:  05 

tTC  . . . 

£MX-iai 

£TC  . . . 

£MX^j.Q5 

cTC  ... 


£MX.i22 

ETC  ... 


ETC 

£MX-li2 

ETC cNCLUSUrtE  (2) 

E-4 


& 


PASU  STATUS  BY  ACAT 


ACAT  1 

CQUt  NP  PHQOS  Ci  PASU  SKcD  RfcQUfcbTtO  GHANTfcU  PtHChNf 


U75 

1 

1 ‘ 

1 

T 

100.0 

03 

1 

0 

0 

0 

00.0 

05 

1 

0 

0 

0 

00.0 

PrtX  103 

1 

0 

0 

0 

00.0 

PMX  105 

3 

1 

1 

1 

100.0 

PMX  122 

2 

0 

0 

0 

00.0 

PSCT  136 

2 

1 

1 

0 

00.0 

PMX  142 

5 

1 

1 

1 

100.0 

TJIAL 

16 

4 

4 

3 

75.0 

icar_ii 


ETC 


ACAl^iil 

ETC  . . • , 


A£Al_lk 

ETC  ... 


I 


I 


ENCLOSURE  (3) 


h»ASU  STATUS  BY  UhUANiZATIuN 


COUc  09 o 

AC  AT  NH  Pt^UUS  CY  f>ASU  bKeP  rttQUfcSIfcJ  GRAMThP  RhhC£NT 

'I  I II  1 100.0 

II  0 0 0 0 00.0 

III  1 0 0 0 00.0 

Iv  10  0 0 00.0 

ruTAL  3 I I I IjOO.O 

coot  03 

ACAf  CY  RASU  bKtO  RhUUtblhD  ORAi^T£D  RckCfc^T 

I 1000  00.0 

II  10  0 0 00.0 

Hi  2 0 0 0 00.0 

IV  4 0 0 0 00.0 

.TuiAi a._  a 0 a oo.o  _ 

cTC 

ETC 

_____  - 

ETC 

EiAX_i22  

ETC 


ETC 

EMA-iiE 

ETC ENCLUSUHc  (4) 


APPENDIX  F 


PRUM* 

NAVDfcP  Tac  COORDINATOR  (CuDc.  OIT) 

30  SEPTfcMBER  197o 

ru« 

DISTRIBUTION  LIST 

SUbJ* 

Program  MiLESTONt  listing 

REP« 

(A)  NAVDEpINST  3960. 4 )A 

tNCL*  (I)  LIST  RtMAIMNU  PRUUkAM  mILcSiJNcS 

(2)  LIST  Or  PHtVlUUSLV  CuMPLHltD  MiLtsTONtS 

1.  IN  ACCQHUANCt  WITH  HErthHNCt  (A),  A PRUURAM  MILcSIONt 
LISTING  IS  PROVIDcD  PJH  THE  rGLLONlNG  PKUGRAm* 


ThIN*  3d9 

TiTLhi  SUBMERSIBLE  SATELLITt  SYSTEM 

PE«  nidSN 

ACAX«  I 

uHG*  PMX  103 


I  - QVERDUE  MILESTONES 

UAIc - EiEIil 

0»  SfcP  7B  - UPDATE  ILS  PLAN 

03  Stp  7o  - COMPLETfc  DT-I  AT  SEA  OtMU 

04  SfcP  la  - START  OT-I 

08  SfcP  7b  - STAhT  DT-I  TEST  REPORT 
:3  sEP  7b  - UPDATE  HDCP 


" milestones  rescheduled  SINoE  LAST  REPUkT 

DATfc EVENT 

02  JUL  78  - PINAL  QI-I  TEST  PLAN  DUE 
28  JUL  7b  - START  UT-I  AT  SEA  DtMO 

III  - MlLfcSTONtS  DUfc  NlThlN  NeXT  THhfct:  MONTHS 


kyjiai 


01  OCT  7b  - DRApT  DT-I  TEST  RePuRT  DUE 


03  OCT  7b  - UPDATE  PROGRAM  MANAGEMENT  PLAN  (PMP) 

07  OCT  78  - UPDATE  DECISION  CUOrtDINATiNG  PAptR  (DCP) 
15  OCT  76  - PiNAL  DT-I  TEST  REPORT  DUE 

22.  OCT  7b  - UPDATE  TAE  MASTER  PLAN  (TEMP) 

02  NoV  lb  - COMPLETE  OT-I 

04  NO  7 7b  - START  QT-I  TEST  HtPURT 

1 1 NOV  78  - (N)SARC  I 

lb  NOV  78  - DRApT  01- I TEST  REPORT  DUE 

29  NOV  7o  - PINAL  UT-I  TEST  RePURT  DUE 

1 1 PtC  76  - COMPLETfc  BrASSBOARD 


F-1 


Ill  - MlLhaiam^Qiifc-dllhLN  HlU  Iritifab  fllbilna  (GUNI) 


UAic  cVhNT 


12  DcC  /b  - CQMi^Lcrc  SUUWCH  bcLtCTIuN 


14 

UhC 

/o 

- 

DSAhC  I 

2d 

UhC 

7d 

STANT  DT-II  TEST  PLAN 

IV 

- MlLcSTuNtS  DJn  IN  POLLbrtlNo  TrtcLVh  MJNlMS 

DATb 

h VEnI 

10 

JAN 

79 

DcMUNSTHAriUN/VALlDATIiJN  CONTNACT  AmAHD 

23 

JAR 

79 

— 

DRAET  DT-II  TeSl  PLAN  DUc 

10 

rhB 

79 

- 

DELIVEH  ADVANCED  DtVtLOPMhNT  MODEL  (ADM) 

Fd 

bed 

79 

— 

final  DT-II  TEST  PLAN  DUE 

03 

MAN 

79 

- 

START  D'i-IlA 

24 

man 

79 

- 

START  OT-II  TEST  PLAN 

15 

APN 

79 

- 

DRAl-T  OT-II  TEST  PLAN  DDE 

01 

MAY 

79 

— 

blNAL  01- 1 1 ThST  PLAN  DUE 

12 

MAY 

79 

- 

START  SYSTEM  RECjUIREMENT  REVIE7(  (SRrt) 

03 

JUN 

79 

- 

COMPLETE  system  REQUIREMENT  kEVIEN  (SRR) 

10 

JUL 

79 

- 

complete  DT-llA 

14 

JUL 

79 

— 

START  SYSTEM  DESIGN  REVIh«  (SDR) 

14 

AUU 

79 

- 

START  OT-II A 

01 

SEP 

79 

— 

complete  OT-II a 

03 

bcp 

79 

- 

START  DT-IId 

01 

UCT 

79 

— 

UPDATE  PROGRAM  MANAGEMENT  PLAN  (PMP) 

21 

ucr 

79 

- 

COMPLETE  DT-IIB 

22 

□CT 

79 

— 

complete  system  design  review  (SDR) 

23 

OCT 

79 

- 

START  DT-II  TEST  REPORT 

01 

NOV 

79 

— 

DRaET  Dl-II  TcSl  RcPORT  DUE 

Od 

NOV 

79 

- 

STArtT  or- 1 Id 

Id 

NOV 

79 

— 

FINAL  DT-II  TEST-r^EPORT  DUE 

01 

DhC 

79 

- 

COMPLETE  SOURCE  ScLtCTION  PLAN 

15 

DEC 

79 

— 

COMPLETE  OT-IIb 

Id 

DhC 

79 

- 

START  OT-II  TEST  REPORT 

03 

JAN 

60 

— 

DRAFT  OT-II  TEST  REPORT  DUe 

15 

JAN 

dO 

ISSUE  RhOUEST  rOR  PROPOSAL  (RFP) 

I.  M. 


UISTHIdUTIUN* 


00 

09 

096 

03 

05 


PMX  103 
PMX  105 
HMX  122 
PMX  136 
PMX  142 
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^ ” LlSI^h  kfaMAl.'^ING  PkOiJiiAM  MILtb'iUNtb 
DA'ft tVetff 7^ 


20  Jan  80  - FINAL  OT-II  TFST  WEFOHT  UUt 

16  hta  80  - UPDATE  iNThOHATED  LOGISTICS  SUPPORT  PLAN  (ILSP) 
22  rhB  80  - ISSUE  DtVELOPvJENT  SpECIpICATION 

02  MAH  80  - start  DT-IIlA  TtST  PLAN 

28  MATi  80  - UPDATE  NA^Y  DECISION  COuRDlNAflNG  PAPtH  (NuCP) 
01  APR  80  - UPDATE  DECISION  COORDINATING  PAPcH  (DCP) 

13  APR  oO  - UPDATE  T&E  MASTER  PLAN  (TEMP) 

14  APR  80  - DRApT  DT-IIIA  TEST  PLAN  DUE 

15  MAY  80  - PINAL  Dl-IIlA  TEST  PLAN  DUE 
30  may  80  - (N)SARC  II 

15  JUN  80  - DSARC  1 1 

20  JUN  60  - complete  SOURCE  SELcCTION 

17  JUN  80  - START  DT-IIlA 


01  JUL  80  - PULL  SCALE  ENGINEERING  DeY  (pSED)  CONTRACT  ARARU 
20  JUL  80  - DELIVER  ENGINEERING  DEVtLQPMENT  MODEL  (cDM) 

01  OCT  80  - UPDATE  PROGRAM  MANAGEMENT  PLAN  (PMP) 

04  OCT  80  - COMPLETE  DT-IIOf 

10  DEC  80  - START  OT-IIlA  TEST  PLAN 

14  Jan  81  - DRAFT  OT-IIlA  TEST  PLAN  DUE 

18  Pc8  81  - riNAL  UT-IIlA  TEST  PLAN  DUE 

22  MAR  81  - COMPLETE  DT-IIlA 

18  APR  8l  - STAriT  OT-IIIA 

10  JUN  ST  - preliminary  DESIgN  REVIEr  (PDR) 

13  JUL  81  - complete  Or-IIIA 

02  ScP  81  - START  DT-IUb  (TECriEVAL)  TEST  PLAN 

08  OCT  81  - DRApT  DT-IIIB  (TECriEVAL)  TEST  PLAN  DUE 

15  NOV  81  - FINAL  DT-IIIb  (TECriEVAL)  TEST  PLAN  DUE 

14  DcC  81  - START  OT-IIIb  (OPEVAL)  TcST  PLAN 

10  JAN  82  - START  DT-IHb  (TECriEVAL) 

lb  JAN  o2  - DRArT  01-1 11b  (LPEYAL)  TcST  PLAN  DUc 

22  rEB  82  - FINAL  OT-IIlB  (OPEVAL)  TEST  PLAN  DUc 

15  MAH  62  - ISSUE  PRODUCT  SpeCIp ICAl iON 

15  MAR  82  - ISSUE  MATERIAL  SPECIp ICATION 

15  MAR  82  - ISSUE  PROCESS  SPcCIp iCAl iON 
01  APS  82  - complete  DT-IIIb  (TeCHEVAL) 

03  APR  82  - START  DT-IIlB  (TcCHeVAL)  TEST  RcPORT 

09  MAY  82  - DRAFT  DT-IIIb  (TECritVAL)  TEST  REPORT  DUe 

12  MAY  82  - REQUESr  FOR  CERTIFICATION  Op  READINESS  rOR  OPEVAL 

10  JUN  82  - CRITICAL  DcSiGN  REVlErt  (CDR) 

16  JUN  82  - PINAL  DT-IllB  (TECricVAL)  TEST  REPORT  DUc 
22  JUL  82  - START  GT-IIIB  (OPEVAL) 

01  OCT  82  - UPDATE  PROGRAM  MANAGEMcnT  PLAN  (PMP) 

04  OCT  82  - COMPLETE  JT-IIIB  (OPEVAL) 

15  NOV  82  - QUICKLOOK  REPORT  DUE 

14  Dec  82  - FINAL  OT-HIB  (OPEVAL)  TEST  REPORT  DUE 


ENCLOSURE  ( 1 ) 
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V - LIST  rtHMAlNiNG  PhUGHAM  MiLLSTuNbS  (CunT) 


DATE  ^ EVENT 


01  JAN  d3  - ktOUhST  FOH  AFFKUVAL  hUH  ShNViCfc  USt  (ASU) 

14  JAN  b3  - FUNCTIONAL  CONf IGUHATION  AUDIT  C^CA) 

28  FtB  83  - APPHOVAL  FUri  StKViChUUSc  (ASU)  GKANTcU 

30  MAN  83  - UPDAT«i  T&fe'  MSIbR  PLAN  deMP)  . 

10  APR  83  - UPDATt  INTtGRATcD  LuGISl'iCS  SUPPORT  PLAN  (ILSP) 

20  APR  83  - UPDATt  NAVY  DtCISION  COORDINATING  PAPER  (NDCP) 

15  may  83  - (N)SARC  III 

20  MAY  83  - update  DECISION  COOhDiNATING  PAPER  (DCP) 

10  JUN  o3  - DSARC  III 

2Q  JUN  83  - PRc-PauDUCHQN  CONTRACT  ArtARD  

30  NOV  83  - (N)SARC  IIlA 

15  DEC  83  - DSARC  II lA 

03  JUL  84  _ PHYSICAL  CONFIGURATION  AUDIT  (DCA) 

12  OCT  84  - (N)SARC  lUb 

16  NOV  84  - DSARC  III3 

25  DEC  84  - PRODUCTION  CONTRACT  ANARD. 

01  JUN  86  - IOC 

13  ttti  87  - FOC 


enclosure  ( 1 ) 


“ list  Or-  PREVIOUSLY  CJM^LfaTfaU  MILfci^YuNES 

tV^______ ^ 

01  JUL  77  - MILhSTONE  ZERJ  - RRUGRAM  INITIATION 
01  SEP  77  - ISSUE  systems  ENGlNtEUlNG  MoT  FLAN  (SEMP) 

01  OCT  77  TISSUE  PROGRAM  MANAGEMENT  PLAN  (PMP) 

25  OCT  77  - START  DT-I  TEST  PLAN 

03  NOV  77  - ISSUE  INITIAL  T&E  MASTER  PLAN  (TEMP) 

01  DEC  77  - ISSUE  SYSTEM  SPEC Ir ICATIOin 

13  DEC  77  - ISSUE  INITIAL  ITEGRATED  LOGlSTlCo  SUPPORT  PLAN 

05  JAN  78  - STAkT  BRASSBOARU 

05  JAN  78  - START  DT-I  LA3  DEMO 

10  JAN  78  - DRApT  DT-I  TEST  PLAN  DUE 

T4-TEB  76  “-  START  DT-  T TEST  PLAN 

15  MAR  7b  - rlNAL  DT-I  TEST  PLAN  DUE 

16  MAR  78  - COMPLETE  DT-I  LAB  DEMO 
25  MAY  78  - DRApT  OT-I  TEST  PLAN  DUE 


Enclosure  (2) 
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APPENDIX  G 


01  i'luVcMotri  197a 


rriUMi  NAVDhh  T&h  CGuRDlNATOk  (CUDc  01 T) 

!□»  OISTiiibUTIQN  LIST 

SUbJi  RukGRT  Jt-  DSAkC  MILb'STOWhS 

(AT  tiA¥£>tok4N5T  3960 ,4 » A 

I*  IN  ACCukDANCt  rtITH  ktrckENCt  (A),  Tht  EuLLUaING  kckukX  Or 
USAriC  MILESTONES  IS  PROVIDED  hOk  THE  MONTH  Or  OCTObEk  I97b» 


DSAkC  I 

TEIN 

DA  It 

ORG 

T ITLE 

PE 

16  DEC  77 

PMX 

103 

SUBMERSIBLE  SATELLITE  SYSTcM 

1 1 1 86N 

389 

06  rEB  7b 

PMX 

122 

OVER  THE  FENCE  RADAR 

32556N 

56B 

16  JDL  78 

05 

ajiomaieu  cofpce  urn 

72690N 

Bo3 

DSAkC  II 

DATE 

ORG 

TIILc  _ 

Pc 

TEiiv 

14  MAk  7b 

03 

WHISTLE  THISTLE  MISSILE 

6245aN 

461 

02  Jan  79 

PMX 

142 

LON  FLYING  SUBMAkiNt  HULL 

6lb22N 

109 

17  xVOV  79 

PMX 

103 

SUBMERSIBLE  SATELLITE  SYSTEM 

1 1 ld6N 

389 

D5ARC  ril 

Uky. 

TITLE  . . _ _ 

£&  __ 

TcIiX 

la  MAY  79 

03 

WHISTLE  TrilbTLE  MISSILE 

6245dN 

461 

1 / AUG  79 

PMX 

139 

MODULAR  GARbAGc  GRlNDcH 

56b97N 

24  J 

I.  M.  SmlrT 


DISTkIBUTlON* 

00  PMX  103 
09  PMX  105 
096  PMX  122 
03  PMX  136 
05  PMX  142 
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01  MAriCH  I97d 


-rtiek*--fMVL>fc^-Tihr  <CoDt  GI-F7  - 

TO*  DlSTrilBUTlON  LIST 

5UbJ«  rfhPURT  Qh  UPtiVAL  MiLtSTUNtS 

KhM  (A)  NAVCtrINST  3960.4  I A 

1.  IN  ACCOrtDANCE  irilH  ktFhriENCt  (A),  TMt  EuLLUnIN'J  HEPuRf  UB 
□PcVAL  MlLESTQNcS  IS  PROVIDED* 


□RU 

IlTLt 

AC  AT 

_Pc 

COMMENC 

C JMPLETc 

03 

XXX  xxxxx  xxxxxx 

I 

1 n22N 

01  JAN 

7d 

03 

APR  16 

03 

05 

X XXX  xxxxxxxxxx 
NONE 

II 

60079N 

14  JUL 

7a 

la 

aEP  16 

PMX 

103 

XXXXX  xxxxxx  XXX 

I 

35749N 

23  SEP 

m 

03 

DEC  /a 

PMX 

105 

XXXXXXX  xx  XXXXX 

III 

7J970N 

14  acT 

16 

16 

JAN  79 

tTC 


I.  M.  OrtlET 


DISTRIBUTION* 

uu 

09 

PMX 

PMX 

103 

105 

096 

03 

PMX 

PMX 

122 

1 36 

**+  SIMILAR  l-ORMAT  EOR  TECHEVAL  REPORTS  *** 

05 

PMX 

142 

r 
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'3  CL 

L/C^tliADizH  IV/0 

PROM* 

NAVDEP  TAb  CJuRDINArUk  (CUUc  01 T) 

TU* 

DISTklBUTION  LIST 

SUbJ*  HhHUHT  Oh  APPt^OVAL  rQk  SckViCfc  U6t  (At>U)  MlLtbTUNtb 
kW-A  (A)  MAVOJBii444Sl  .J960^iA 

1.  1.-4  ACCOkDANCE  WITH  fihbhkhNCt  (A),  THfc  fULLOrtlNO  kEPuHT  ub 


i _ 

090 

03 

PMX  103 
PMX  105 


Puk  SEkVTCc  USE 

(ASU) 

milestones 

Ib 

PkuVIOcD* 

TITLE 

AC  AT 

Pe 

*7  Kl 

nATi-' 

XXX  XXXXXX  XXX  X 
AA  AAAAA  AAAAAAA 

KRRRRR  RRRRR  RRR 

I 

HI 

I I 

AA 

f AM 

IQ 

i Vi 

25447N 

’<  -iAM 

16 

1 

yj  Ari 

FEb 

U A u 

1 y 

79 

70 

DDDDDD  wOODO  DOO 

XXXXXXXXX  XXXXXX 

1 i 

I 

O OwM 

32557N 

15 

MArt 

MAY 

r y 

79 

ETC 


1.  M.  Swi^T 


Uli^THloUTI-ON* 

UU 

09 

096 

03 

05 

PMX  103 
PMX  105 
PMX  122 
PMX  136 
PMX  142 
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OJ  otPftMbfc:.i  197a 


PrtUM* 

roi 

NAVDtP  T&t  CGUROINATGH  (CUDh  OIT) 
UISTklbUTlGN  LIST 

subJ* 

kcPGHf  Gb  DVhKDJh  MiLhS TONES 

Ptp* 

(A)  NAVOEPINST  3960.4 lA 

1.  IM  ACCUKDANCE  n'lTH  RtPhHtMCt  (A),  Thh 
JVcHDUt  MlLtSrUNHS  IS  PHGVlDEJ* 

puLLJaING  PEPukT  up 

GfiO 

ThIN  PE  ACAT  EVbNT 

- uATt  .. 

09o 

NONE 

03  NONb 


05 

PMX 

103 

465 
5b  7 

496 /2N 
49572N 

1 

II 

UPDATE  TEMP 

PINAL  CT-II  TEST 

PLN 

03 

Id 

OCT 

JUL 

7/ 

7d 

PMX 

105 

779 

aa905N 

IV 

COMMENCE  01- I 

— 

21 

JUL 

78 

PMX 

122 

634 

1 1325N 

III 

CRITICAL  Design 

REV 

25 

JUL 

7d 

PMX  136  NuNt 

cTC  


I . M.  SrtIf  T 


OlSTPlbUTfUN* 

00 

09 

096 

03 

05 

PMX  103 
PMX  105 
PMX  122 
PMX  136 
PMX  142 
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APPENDIX  K 


05  fcBHUAKY  I97d 


t-tiUMt  NAVDth  T&h  CObROlNATUk  (COUt  OIT) 

TG»  DISTHIBUTIOM 

SUBJ*  TfcMP/T£P  UPDATE  STATUS  REPORT 

T . BASED  UN  THE  REQTJ IREMENT  TO  “OPDATfc  TE'MPSVTEPS  OR ' AN 
ANNUAL  BASIS  AND  ALSO  TAU  MONTHS  PRIOR  TO  MAJOR  DECISION 
POINTS,  THE  FOLLOWING  DOCUMENTS  WILL  REQUIRE  UPDATING  DUftiNo 
THc  NEXT  TrttLVE  MONTHS* 


TEMPS  ( ACATS  I , I f ANlTTITr 


PRO TUN  AC^T  L^Tl REASON 

OS  -657  in  06  MAR  7«  ANNUAL 


PMX 

142 

346 

II 

03 

AUG 

73 

ANNUAL 

PMX 

103 

1 13 

1 

14 

SEP 

7b 

DSAftC  1 I 

096 

104 

I 

30 

NOV 

7b 

NSARC  I 

05 

677 

I 

24 

APR 

79 

OSARC  I 

TEPS  (ACAT  IV) 


PRO 

IklN 

DATE 

reason 

03  ' 

576 

22  FEB  79 

ANNUAL 

PMX 

105 

598 

03  MAY  79 

ANNUAL 

PMX 

122 

391 

29  JUN  79 

ANNUAL 

I.  M.  SWIFT 


I 

1 

i 


00 
09 
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03 
05  ~ 
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